Oct 31 03:25:25.128503 ip-10-0-134-106 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 03:25:25.128518 ip-10-0-134-106 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 03:25:25.128528 ip-10-0-134-106 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 03:25:25.128929 ip-10-0-134-106 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 03:25:35.226511 ip-10-0-134-106 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 03:25:35.226530 ip-10-0-134-106 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7839c4b092a94f4e84f3928144bc5271 -- Oct 31 03:27:41.478937 ip-10-0-134-106 systemd[1]: Starting Kubernetes Kubelet... Oct 31 03:27:41.914323 ip-10-0-134-106 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:27:41.914323 ip-10-0-134-106 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 03:27:41.914323 ip-10-0-134-106 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:27:41.914323 ip-10-0-134-106 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:27:41.923227 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923126 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:27:41.923227 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923226 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:27:41.923227 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923230 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923234 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923237 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923240 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923243 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923246 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923250 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923252 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923255 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923258 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923260 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923263 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923266 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923268 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923271 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923274 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923276 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923279 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923281 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923284 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:27:41.923403 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923287 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923289 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923292 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923298 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923301 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923304 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923306 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923309 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923311 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923314 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923317 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923320 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923323 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923327 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923332 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923335 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923338 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923341 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923344 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:27:41.923882 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923346 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923349 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923352 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923354 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923357 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923359 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923362 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923364 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923368 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923370 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923373 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923375 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923378 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923382 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923384 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923387 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923389 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923392 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923394 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:27:41.924336 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923398 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923402 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923405 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923409 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923412 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923414 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923418 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923421 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923424 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923427 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923430 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923432 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923435 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923438 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923440 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923443 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923445 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923448 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923451 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:27:41.924794 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923454 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923456 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923458 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923462 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923464 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923467 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923471 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923473 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923476 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923478 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923481 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923483 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923486 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923488 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923491 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923493 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923495 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923498 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923500 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:27:41.925247 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923503 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923506 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923509 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923512 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923514 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923517 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923520 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.923523 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925234 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925243 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925246 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925250 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925253 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925256 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925259 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925262 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925265 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925269 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925272 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925275 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:27:41.925739 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925277 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925280 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925283 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925285 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925288 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925290 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925294 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925297 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925300 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925302 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925305 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925307 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925310 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925313 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925316 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925319 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925322 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925324 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925327 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925329 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:27:41.926214 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925332 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925335 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925337 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925340 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925342 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925345 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925347 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925350 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925352 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925354 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925357 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925360 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925363 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925365 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925368 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925370 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925373 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925375 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925378 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:27:41.926701 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925382 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925384 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925387 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925390 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925394 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925397 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925400 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925403 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925406 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925409 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925412 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925415 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925418 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925420 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925423 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925425 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925428 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925430 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925432 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:27:41.927154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925435 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925438 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925440 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925443 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925445 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925449 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925451 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925454 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925456 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925459 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925461 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925464 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925467 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925470 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925472 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925475 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925477 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925480 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925483 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925487 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:27:41.927606 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925490 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925493 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925496 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925499 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925502 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925504 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925506 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925509 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925511 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925514 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925517 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925519 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925522 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925524 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925526 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.925529 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926042 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926048 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926051 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:27:41.928082 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926054 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926057 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926059 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926062 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926064 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926068 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926070 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926073 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926076 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926078 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926081 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926084 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926087 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926089 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926092 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926095 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926097 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926100 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926102 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:27:41.928527 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926105 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926108 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926112 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926115 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926118 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926121 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926124 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926126 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926129 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926132 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926134 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926136 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926140 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926142 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926145 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926147 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926151 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926153 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926156 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926159 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:27:41.928998 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926161 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926164 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926168 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926171 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926175 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926177 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926180 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926182 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926185 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926188 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926191 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926193 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926195 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926198 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926201 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926203 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926206 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926208 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926211 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926213 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:27:41.929465 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926216 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926219 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926221 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926224 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926227 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926230 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926233 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926235 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926238 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926241 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926243 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926246 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926249 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926251 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926254 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926257 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926259 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926262 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926264 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926267 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:27:41.929942 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926270 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926272 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926275 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926278 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926281 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926283 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926285 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926288 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926290 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926293 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926295 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926298 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926300 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926303 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926305 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926308 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926310 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926312 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926315 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:27:41.930455 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926318 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926321 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926323 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926329 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.926332 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927469 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927480 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927487 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927492 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927497 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927501 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927505 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927509 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927513 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927517 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927521 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927525 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927528 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927531 2576 flags.go:64] FLAG: --cgroup-root="" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927534 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927537 2576 flags.go:64] FLAG: --client-ca-file="" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927540 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927544 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927549 2576 flags.go:64] FLAG: --cluster-domain="" Oct 31 03:27:41.930916 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927552 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927555 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927558 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927562 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927567 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927570 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927573 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927576 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927580 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927584 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927587 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927591 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927594 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927599 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927602 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927605 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927608 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927612 2576 flags.go:64] FLAG: --enable-server="true" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927615 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927619 2576 flags.go:64] FLAG: --event-burst="100" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927622 2576 flags.go:64] FLAG: --event-qps="50" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927637 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927641 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927644 2576 flags.go:64] FLAG: --eviction-hard="" Oct 31 03:27:41.931481 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927649 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927652 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927655 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927658 2576 flags.go:64] FLAG: --eviction-soft="" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927662 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927665 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927668 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927671 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927674 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927677 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927680 2576 flags.go:64] FLAG: --feature-gates="" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927684 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927687 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927691 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927694 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927697 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927700 2576 flags.go:64] FLAG: --help="false" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927704 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927709 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927712 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927716 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927719 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927723 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 03:27:41.932084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927726 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927729 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927732 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927735 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927738 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927741 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927744 2576 flags.go:64] FLAG: --kube-reserved="" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927747 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927750 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927753 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927756 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927759 2576 flags.go:64] FLAG: --lock-file="" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927762 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927765 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927769 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927774 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927777 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927780 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927783 2576 flags.go:64] FLAG: --logging-format="text" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927787 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927790 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927793 2576 flags.go:64] FLAG: --manifest-url="" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927796 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927801 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927805 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 03:27:41.932647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927808 2576 flags.go:64] FLAG: --max-pods="110" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927812 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927815 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927819 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927822 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927826 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927829 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927832 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927840 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927843 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927846 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927850 2576 flags.go:64] FLAG: --pod-cidr="" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927853 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927856 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927859 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927862 2576 flags.go:64] FLAG: --port="10250" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927865 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927868 2576 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0b963bd819d464b75" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927871 2576 flags.go:64] FLAG: --qos-reserved="" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927874 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927878 2576 flags.go:64] FLAG: --register-node="true" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927881 2576 flags.go:64] FLAG: --register-with-taints="" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927885 2576 flags.go:64] FLAG: --registry-burst="10" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927888 2576 flags.go:64] FLAG: --registry-qps="5" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927891 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927894 2576 flags.go:64] FLAG: --reserved-memory="" Oct 31 03:27:41.933239 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927898 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927901 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927904 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927907 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927910 2576 flags.go:64] FLAG: --runonce="false" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927913 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927917 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927920 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927923 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927927 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927930 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927934 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927937 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927940 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927943 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927947 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927950 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927953 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927956 2576 flags.go:64] FLAG: --system-cgroups="" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927959 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927965 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927968 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927971 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927975 2576 flags.go:64] FLAG: --tls-min-version="" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927978 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 03:27:41.933909 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927981 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927984 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927987 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927990 2576 flags.go:64] FLAG: --v="2" Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927995 2576 flags.go:64] FLAG: --version="false" Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.927999 2576 flags.go:64] FLAG: --vmodule="" Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.928003 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.928007 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928171 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928175 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928178 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928181 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928183 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928186 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928189 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928191 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928194 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928198 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928201 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928203 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928206 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:27:41.934498 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928208 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928212 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928214 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928217 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928220 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928222 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928225 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928227 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928230 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928232 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928235 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928238 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928240 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928243 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928246 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928249 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928251 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928254 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928257 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928261 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:27:41.935002 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928264 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928266 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928269 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928272 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928275 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928277 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928280 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928284 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928287 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928291 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928293 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928296 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928298 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928301 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928304 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928306 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928309 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928311 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928314 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:27:41.935480 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928316 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928319 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928322 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928326 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928330 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928334 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928337 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928339 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928342 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928345 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928347 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928352 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928356 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928359 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928362 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928364 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928367 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928370 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928372 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928375 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:27:41.935973 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928378 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928380 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928383 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928386 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928388 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928391 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928393 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928396 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928398 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928401 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928404 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928406 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928408 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928411 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928413 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928416 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928418 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928421 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928423 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:27:41.936750 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928426 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928428 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928431 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928434 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928438 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928442 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928444 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928447 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928450 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928452 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928455 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928458 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928460 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928463 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928466 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.928494 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:27:41.937415 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928532 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928537 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928541 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928544 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928548 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928551 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928553 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928556 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928559 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928561 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928564 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928566 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928569 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928572 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928574 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928578 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928580 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928583 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928585 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:27:41.937953 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928588 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928592 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928594 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928598 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928601 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928603 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928606 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928609 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928611 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928614 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928616 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928620 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928624 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928637 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928641 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928644 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928647 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928658 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928661 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:27:41.938413 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928664 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928667 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928669 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928672 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928675 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928677 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928680 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928685 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928688 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928691 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928694 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928697 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928699 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928702 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928706 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928709 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928712 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928715 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928718 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928720 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:27:41.938885 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928722 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928725 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928728 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928730 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928733 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928736 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928738 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928741 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928743 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928747 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928750 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928752 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928755 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928757 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928760 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928763 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928765 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928767 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928770 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928772 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:27:41.939348 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928775 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928778 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928781 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928783 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928785 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928788 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928792 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928794 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928799 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928801 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928804 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928806 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928809 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928812 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928814 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928817 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928820 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928823 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928826 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:27:41.939909 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928828 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928831 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928833 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928836 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928839 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928841 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928844 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928846 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.928849 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.938125 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.938142 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.938911 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.938918 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.938977 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.938983 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.938986 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.938989 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.938992 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.938995 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.938997 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:27:41.940388 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939000 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939003 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939006 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939009 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939012 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939014 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939017 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939019 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939023 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939027 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939031 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939034 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939036 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939039 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939042 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939045 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939048 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939051 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939053 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:27:41.940883 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939056 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939059 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939062 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939065 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939068 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939071 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939074 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939077 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939079 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939082 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939085 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939087 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939090 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939093 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939096 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939098 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939101 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939103 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939106 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939108 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:27:41.941335 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939111 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939113 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939116 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939119 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939121 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939124 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939126 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939129 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939132 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939134 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939137 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939139 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939142 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939144 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939147 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939150 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939153 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939156 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939158 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939161 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:27:41.941833 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939163 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939166 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939169 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939171 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939174 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939176 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939179 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939181 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939184 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939187 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939190 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939192 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939195 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939199 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939201 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939204 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939206 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939209 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939211 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:27:41.942309 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939214 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939217 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939221 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939224 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939228 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939231 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939234 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939237 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939240 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939242 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939245 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939248 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939250 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939253 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939255 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939258 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939260 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939263 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939265 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:27:41.942766 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939268 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939271 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.939301 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939484 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939492 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939497 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939500 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939503 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939507 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939510 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939513 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939516 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939518 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939521 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939524 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:27:41.943219 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939526 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939529 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939532 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939534 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939537 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939540 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939542 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939545 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939548 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939550 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939553 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939555 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939558 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939561 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939563 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939566 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939568 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939571 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939573 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939576 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:27:41.943581 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939578 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939581 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939583 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939586 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939588 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939591 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939594 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939597 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939600 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939603 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939605 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939608 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939611 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939613 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939616 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939618 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939621 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939624 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939643 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939647 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:27:41.944066 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939649 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939652 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939655 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939657 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939660 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939662 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939665 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939668 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939670 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939673 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939675 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939678 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939680 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939683 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939685 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939688 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939690 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939694 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939697 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939701 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:27:41.944534 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939704 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939707 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939709 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939712 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939714 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939717 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939720 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939723 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939726 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939729 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939731 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939734 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939737 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939739 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939742 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939744 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939747 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939750 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939752 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:27:41.945019 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939755 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939757 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939760 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939762 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939765 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939767 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939770 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939772 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939775 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939777 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939780 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939782 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939785 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939788 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939790 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.939816 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:27:41.945456 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939852 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939856 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939859 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939862 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939865 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939868 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939870 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939873 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939876 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939878 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939881 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939883 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939886 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939888 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939891 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939893 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939896 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939898 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939901 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939904 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:27:41.945843 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939906 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939908 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939911 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939915 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939920 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939923 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939925 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939928 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939930 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939933 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939937 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939939 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939942 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939945 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939947 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939950 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939952 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939955 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939957 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:27:41.946328 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939960 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939962 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939965 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939968 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939970 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939973 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939975 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939977 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939980 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939983 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939985 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939988 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939990 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939992 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939995 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.939997 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940000 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940002 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940005 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940007 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:27:41.946779 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940010 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940012 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940017 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940021 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940024 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940028 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940031 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940034 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940036 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940038 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940041 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940044 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940047 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940050 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940052 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940055 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940057 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940060 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940062 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:27:41.947241 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940065 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940067 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940070 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940073 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940075 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940078 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940080 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940083 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940085 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940088 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940090 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940093 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940095 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940098 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940100 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940103 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940106 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940108 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940112 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940115 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:27:41.947871 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940117 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940120 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940122 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940125 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940128 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940130 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940134 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:41.940136 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.940198 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.942723 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.943798 2576 server.go:1014] "Starting client certificate rotation" Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.943899 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.943943 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.946118 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:27:41.948330 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.947247 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:27:41.955813 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.955794 2576 log.go:25] "Validated CRI v1 runtime API" Oct 31 03:27:41.959948 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.959933 2576 log.go:25] "Validated CRI v1 image API" Oct 31 03:27:41.960763 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.960732 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 03:27:41.967688 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.967666 2576 fs.go:135] Filesystem UUIDs: map[2a199e6e-6ab4-4b13-91c4-0f5097e5af7d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cf1e70e4-adaa-4c97-a7a0-211a11de3f22:/dev/nvme0n1p3] Oct 31 03:27:41.967768 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.967687 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 03:27:41.974709 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.974567 2576 manager.go:217] Machine: {Timestamp:2026-10-31 03:27:41.972012267 +0000 UTC m=+0.374625622 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102721 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec217aa3ba86f35ed6ebe6fda6cd842c SystemUUID:ec217aa3-ba86-f35e-d6eb-e6fda6cd842c BootID:7839c4b0-92a9-4f4e-84f3-928144bc5271 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e6:02:f2:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e6:02:f2:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:a3:5a:82:2f:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 03:27:41.974709 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.974700 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 03:27:41.974874 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.974811 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 03:27:41.977861 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.977837 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 03:27:41.978011 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.977862 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-106.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 03:27:41.978061 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.978013 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 03:27:41.978061 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.978021 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 03:27:41.978061 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.978045 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:27:41.978172 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.978063 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:27:41.979408 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.979397 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 03:27:41.984142 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.984130 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 03:27:41.984303 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.984293 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 03:27:41.985179 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.985168 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 31 03:27:41.985214 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.985185 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 03:27:41.985214 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.985195 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 03:27:41.985214 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.985204 2576 kubelet.go:405] "Adding apiserver pod source" Oct 31 03:27:41.985214 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.985213 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 03:27:41.985646 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.985615 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:27:41.989474 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.989452 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 03:27:41.996598 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.996584 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:27:41.997403 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.997391 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 03:27:41.997442 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.997421 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 03:27:41.997580 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.997571 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 03:27:41.997612 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.997587 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 03:27:41.997612 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.997593 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 03:27:41.997612 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.997599 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 03:27:41.997612 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.997604 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 03:27:41.997612 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.997609 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 03:27:41.998401 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.998392 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 03:27:41.998401 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.998401 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 03:27:41.998457 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.998409 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 03:27:41.998457 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.998416 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 03:27:41.998457 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.998425 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 03:27:41.998457 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.998434 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 03:27:41.999933 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.999924 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 03:27:41.999973 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:41.999934 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 03:27:42.003833 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.003819 2576 server.go:1263] "Started kubelet" Oct 31 03:27:42.003968 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.003933 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 03:27:42.003968 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.003916 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 03:27:42.004118 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.003997 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 03:27:42.004597 ip-10-0-134-106 systemd[1]: Started Kubernetes Kubelet. Oct 31 03:27:42.005689 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.005565 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 03:27:42.006856 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.006838 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 31 03:27:42.010018 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.010000 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 03:27:42.013110 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.013086 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 03:27:42.013781 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.013763 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 03:27:42.013890 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.013860 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-106.us-west-2.compute.internal\" not found" Oct 31 03:27:42.014536 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014515 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 03:27:42.014655 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014517 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 03:27:42.014655 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014551 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 03:27:42.014655 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014591 2576 factory.go:55] Registering systemd factory Oct 31 03:27:42.014655 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014606 2576 factory.go:223] Registration of the systemd container factory successfully Oct 31 03:27:42.014655 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014639 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 31 03:27:42.014655 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014650 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 31 03:27:42.014866 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014853 2576 factory.go:153] Registering CRI-O factory Oct 31 03:27:42.014896 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014870 2576 factory.go:223] Registration of the crio container factory successfully Oct 31 03:27:42.014923 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014910 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 03:27:42.014967 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014928 2576 factory.go:103] Registering Raw factory Oct 31 03:27:42.014967 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.014957 2576 manager.go:1199] Started watching for new ooms in manager Oct 31 03:27:42.016219 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.016199 2576 manager.go:319] Starting recovery of all containers Oct 31 03:27:42.029051 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.029019 2576 manager.go:324] Recovery completed Oct 31 03:27:42.034047 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.034030 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:27:42.036089 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.036056 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:27:42.036172 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.036104 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:27:42.036172 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.036116 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:27:42.036774 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.036760 2576 cpu_manager.go:226] "Starting" policy="none" Oct 31 03:27:42.036774 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.036767 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:27:42.036774 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.036772 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 03:27:42.036931 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.036792 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 03:27:42.037450 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.037433 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:27:42.037539 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.037459 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:27:42.037539 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.037469 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:27:42.039036 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.039023 2576 policy_none.go:50] "Start" Oct 31 03:27:42.039036 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.039038 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 03:27:42.039147 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.039048 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 03:27:42.041029 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.041017 2576 policy_none.go:44] "Start" Oct 31 03:27:42.118778 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.114937 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-106.us-west-2.compute.internal\" not found" Oct 31 03:27:42.145561 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.145524 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 03:27:42.146731 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.146711 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 03:27:42.146731 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.146735 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 03:27:42.146877 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.146761 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 03:27:42.146877 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.146800 2576 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 03:27:42.147943 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.147908 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-106.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 03:27:42.148037 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.147873 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-106.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 03:27:42.148584 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.147867 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-106.us-west-2.compute.internal.18e37f7f73ca1e4f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-106.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-106.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 03:27:42.003789391 +0000 UTC m=+0.406402747,LastTimestamp:2026-10-31 03:27:42.003789391 +0000 UTC m=+0.406402747,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-106.us-west-2.compute.internal,}" Oct 31 03:27:42.209269 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.209226 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 03:27:42.211846 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.211832 2576 manager.go:341] "Starting Device Plugin manager" Oct 31 03:27:42.211924 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.211914 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 03:27:42.211924 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.211924 2576 server.go:87] "Starting device plugin registration server" Oct 31 03:27:42.212165 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.212155 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 03:27:42.212318 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.212167 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 03:27:42.212436 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.212353 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 03:27:42.212436 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.212435 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 03:27:42.212543 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.212443 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 03:27:42.213039 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.213021 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 03:27:42.213132 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.213074 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-106.us-west-2.compute.internal\" not found" Oct 31 03:27:42.240746 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.240722 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 03:27:42.240837 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.240774 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g9wqk" Oct 31 03:27:42.247583 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.247565 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal"] Oct 31 03:27:42.247655 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.247621 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:27:42.248568 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.248552 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:27:42.248614 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.248583 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:27:42.248614 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.248593 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:27:42.250823 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.250810 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:27:42.250953 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.250940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.250988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.250970 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:27:42.251763 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.251739 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:27:42.251865 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.251769 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:27:42.251865 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.251781 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:27:42.251865 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.251788 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:27:42.251865 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.251810 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:27:42.251865 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.251824 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:27:42.253866 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.253853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.253933 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.253876 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:27:42.254580 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.254560 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:27:42.254680 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.254587 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:27:42.254680 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.254601 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:27:42.284392 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.284369 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-106.us-west-2.compute.internal\" not found" node="ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.286379 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.286363 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-106.us-west-2.compute.internal\" not found" node="ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.300824 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.300805 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:27:42.302017 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.302001 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:27:42.312985 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.312964 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:27:42.314201 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.314184 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:27:42.314274 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.314216 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:27:42.314274 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.314226 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:27:42.314274 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.314255 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.316386 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.316370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.316443 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.316395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.316443 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.316413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/506bf97362686c30bca7e209bcab60a8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"506bf97362686c30bca7e209bcab60a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.394085 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.394054 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:27:42.394716 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.394696 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.394798 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:42.394731 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-106.us-west-2.compute.internal\": node \"ip-10-0-134-106.us-west-2.compute.internal\" not found" Oct 31 03:27:42.395122 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.395104 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:27:42.414955 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.414928 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.417112 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.417084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.417223 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.417127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.417223 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.417156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/506bf97362686c30bca7e209bcab60a8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"506bf97362686c30bca7e209bcab60a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.417223 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.417183 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.417223 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.417191 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/506bf97362686c30bca7e209bcab60a8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"506bf97362686c30bca7e209bcab60a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.417223 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.417192 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.476506 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.476431 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:27:42.476671 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.476553 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.530940 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.530906 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:27:42.587900 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.587855 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:42.588938 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:42.588922 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:27:44.039273 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.039230 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 03:27:44.039798 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.039392 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 03:27:44.039798 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.039456 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 03:27:44.039798 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.039482 2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aad67609fd0bc481f905aa31524c4d83-10f2963fa6f7f4fa.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-106.us-west-2.compute.internal.18e37f7f75b6fa86\": read tcp 10.0.134.106:51814->3.151.147.33:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-106.us-west-2.compute.internal.18e37f7f75b6fa86 default 4940 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-106.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-106.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-106.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 03:27:42 +0000 UTC,LastTimestamp:2026-10-31 03:27:42.251755345 +0000 UTC m=+0.654368700,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-106.us-west-2.compute.internal,}" Oct 31 03:27:44.082277 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.082240 2576 apiserver.go:52] "Watching apiserver" Oct 31 03:27:44.109695 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.109665 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 03:27:44.330367 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.330338 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 03:27:44.330662 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.330644 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-2vww2","openshift-image-registry/node-ca-fb4pz","openshift-multus/multus-additional-cni-plugins-7txzf","openshift-network-diagnostics/network-check-target-bh9xh","kube-system/konnectivity-agent-zmcjg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428","openshift-cluster-node-tuning-operator/tuned-v5d8b","openshift-network-operator/iptables-alerter-lt92b","kube-system/global-pull-secret-syncer-n9rx6","openshift-ovn-kubernetes/ovnkube-node-4hh72","kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal","openshift-multus/multus-9sx4r","openshift-multus/network-metrics-daemon-rh4rl"] Oct 31 03:27:44.333079 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.333061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2vww2" Oct 31 03:27:44.335152 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.335110 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:27:44.337194 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.337172 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.339257 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.339241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:44.339346 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.339310 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:27:44.341283 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.341263 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:22:43 +0000 UTC" deadline="2028-05-08 05:25:10.619647744 +0000 UTC" Oct 31 03:27:44.341283 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.341282 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13321h57m26.278368182s" Oct 31 03:27:44.341361 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.341288 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:44.343346 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.343333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.345370 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.345357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.347501 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.347478 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:27:44.349582 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.349563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:44.349679 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.349611 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:27:44.351735 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.351713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.354027 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.354008 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.356218 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.356200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:44.356294 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.356258 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:27:44.381968 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.381944 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 03:27:44.381968 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.381949 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 03:27:44.382097 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.382044 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hztlx\"" Oct 31 03:27:44.383398 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.383382 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 03:27:44.383585 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.383569 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 03:27:44.383711 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.383697 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z9sxx\"" Oct 31 03:27:44.383745 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.383701 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 03:27:44.385851 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.385838 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 03:27:44.385940 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.385918 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7b9lc\"" Oct 31 03:27:44.385978 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.385965 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 03:27:44.385978 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.385970 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 03:27:44.386036 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.386011 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 03:27:44.386036 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.386027 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 03:27:44.389784 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.389749 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 03:27:44.410867 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.410845 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 03:27:44.416789 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.416765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.416895 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.416802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-node-log\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.416895 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.416827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-tmp-dir\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:27:44.417013 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.416908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac082af6-5ee0-4995-9636-8271debb950a-host\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:27:44.417013 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.416962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysconfig\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.417232 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-conf-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.417232 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckh87\" (UniqueName: \"kubernetes.io/projected/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-kube-api-access-ckh87\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:44.417232 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-host\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.417383 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-kubelet\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.417383 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-system-cni-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.417383 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dd3e64a-9a74-4542-b09a-7003e749b44f-agent-certs\") pod \"konnectivity-agent-zmcjg\" (UID: \"3dd3e64a-9a74-4542-b09a-7003e749b44f\") " pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:44.417383 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-slash\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.417383 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-etc-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.417629 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-socket-dir-parent\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.417629 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-multus-certs\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.417629 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-netns\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.417629 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq8vh\" (UniqueName: \"kubernetes.io/projected/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-kube-api-access-dq8vh\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:27:44.417629 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-system-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.417629 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-k8s-cni-cncf-io\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.417629 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417614 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-multus\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.417922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-socket-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.417922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-registration-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.417922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.417922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcs67\" (UniqueName: \"kubernetes.io/projected/288945de-f0e7-47c7-87c9-45f8dbc021d9-kube-api-access-xcs67\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.417922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-sys\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.417922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-bin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.417922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-etc-kubernetes\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.417922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:44.418200 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.418200 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.417977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cnibin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.418200 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.418200 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmkwg\" (UniqueName: \"kubernetes.io/projected/fb187718-37b3-47fa-8c44-cd650c0185be-kube-api-access-nmkwg\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.418200 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-kubelet-config\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:44.418200 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-ovn\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.418200 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-lib-modules\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.418200 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.418200 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.418200 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-config\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.418464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-tmp\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.418464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.418464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-daemon-config\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.418464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpjn\" (UniqueName: \"kubernetes.io/projected/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kube-api-access-jkpjn\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.418464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28de4e55-40b4-4721-9f39-2bbd1033c06e-iptables-alerter-script\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:27:44.418464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:44.418464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-systemd-units\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.418464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-var-lib-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.418464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-script-lib\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.418699 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-hosts-file\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:27:44.418699 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgl9x\" (UniqueName: \"kubernetes.io/projected/ac082af6-5ee0-4995-9636-8271debb950a-kube-api-access-kgl9x\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:27:44.418699 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-var-lib-kubelet\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.418699 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.418699 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-log-socket\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.418699 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cni-binary-copy\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.418699 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-device-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.418699 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-kubelet\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.418699 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418697 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-bin\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.418938 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-dbus\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:44.418938 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-systemd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.418938 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-netns\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.418938 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtjc\" (UniqueName: \"kubernetes.io/projected/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-kube-api-access-hrtjc\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.418938 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-cnibin\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.418938 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76gl\" (UniqueName: \"kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl\") pod \"network-check-target-bh9xh\" (UID: \"fc1bac22-a12b-4777-989e-a4c059633fd1\") " pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:44.418938 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-modprobe-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.418938 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-kubernetes\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.419179 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418944 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-conf\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.419179 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.418973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-run\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.419179 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdqlc\" (UniqueName: \"kubernetes.io/projected/f7bf98c9-3458-4936-8dfd-32faa8e522b9-kube-api-access-mdqlc\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.419179 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-binary-copy\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.419179 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdfsd\" (UniqueName: \"kubernetes.io/projected/28de4e55-40b4-4721-9f39-2bbd1033c06e-kube-api-access-zdfsd\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:27:44.419179 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419113 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-netd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.419179 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovn-node-metrics-cert\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.419452 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac082af6-5ee0-4995-9636-8271debb950a-serviceca\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:27:44.419452 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.419452 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-systemd\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.419452 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.419452 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419331 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dd3e64a-9a74-4542-b09a-7003e749b44f-konnectivity-ca\") pod \"konnectivity-agent-zmcjg\" (UID: \"3dd3e64a-9a74-4542-b09a-7003e749b44f\") " pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:44.419452 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-tuned\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.419452 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-hostroot\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.419452 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28de4e55-40b4-4721-9f39-2bbd1033c06e-host-slash\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:27:44.419452 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419449 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-env-overrides\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.419689 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.419495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-sys-fs\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.434044 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.434024 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 03:27:44.434135 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.434062 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 03:27:44.434265 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.434252 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 03:27:44.434321 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.434292 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 03:27:44.434368 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.434346 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-js2xb" Oct 31 03:27:44.434905 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.434886 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:27:44.434968 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.434926 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 03:27:44.435012 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.434969 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:27:44.435062 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435016 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 03:27:44.435062 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435019 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 03:27:44.435062 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435045 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 03:27:44.435207 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435067 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 03:27:44.435207 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435077 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2rc66\"" Oct 31 03:27:44.435207 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435087 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 03:27:44.435207 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435047 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 03:27:44.435207 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435101 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 03:27:44.435207 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435147 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hp5ds\"" Oct 31 03:27:44.435207 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435156 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 03:27:44.435207 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435165 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 03:27:44.435207 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435149 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hp8t5\"" Oct 31 03:27:44.435207 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435208 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dlcmg\"" Oct 31 03:27:44.435526 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435269 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 03:27:44.435526 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435341 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zvcgd\"" Oct 31 03:27:44.435526 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.435397 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8nlcv\"" Oct 31 03:27:44.520537 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-tmp\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.520537 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.520820 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-daemon-config\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.520820 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpjn\" (UniqueName: \"kubernetes.io/projected/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kube-api-access-jkpjn\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.520820 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28de4e55-40b4-4721-9f39-2bbd1033c06e-iptables-alerter-script\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:27:44.520820 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:44.520820 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-systemd-units\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.520820 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-var-lib-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.520820 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-script-lib\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.520820 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520788 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-var-lib-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.520820 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.520766 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520747 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520782 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-systemd-units\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-hosts-file\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgl9x\" (UniqueName: \"kubernetes.io/projected/ac082af6-5ee0-4995-9636-8271debb950a-kube-api-access-kgl9x\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520916 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-var-lib-kubelet\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521000 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-hosts-file\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.520996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-log-socket\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521056 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-log-socket\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cni-binary-copy\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521094 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-var-lib-kubelet\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-device-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521143 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521261 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-device-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521265 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-daemon-config\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.521287 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521290 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28de4e55-40b4-4721-9f39-2bbd1033c06e-iptables-alerter-script\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-kubelet\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.521353 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret podName:6e9ea6af-1755-4e15-861b-5740354972f2 nodeName:}" failed. No retries permitted until 2026-10-31 03:27:45.021291844 +0000 UTC m=+2.328408871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret") pod "global-pull-secret-syncer-n9rx6" (UID: "6e9ea6af-1755-4e15-861b-5740354972f2") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521389 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-kubelet\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-bin\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521426 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-dbus\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-systemd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521492 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-script-lib\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521498 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-bin\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521521 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-dbus\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-netns\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521575 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-systemd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtjc\" (UniqueName: \"kubernetes.io/projected/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-kube-api-access-hrtjc\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cni-binary-copy\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521614 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-netns\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-cnibin\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521663 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-cnibin\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.521946 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76gl\" (UniqueName: \"kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl\") pod \"network-check-target-bh9xh\" (UID: \"fc1bac22-a12b-4777-989e-a4c059633fd1\") " pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-modprobe-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-kubernetes\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521756 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-conf\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521772 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-kubernetes\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-run\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqlc\" (UniqueName: \"kubernetes.io/projected/f7bf98c9-3458-4936-8dfd-32faa8e522b9-kube-api-access-mdqlc\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521833 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-modprobe-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-binary-copy\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521877 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-run\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521922 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-conf\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdfsd\" (UniqueName: \"kubernetes.io/projected/28de4e55-40b4-4721-9f39-2bbd1033c06e-kube-api-access-zdfsd\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.521993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-netd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovn-node-metrics-cert\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac082af6-5ee0-4995-9636-8271debb950a-serviceca\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522086 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-netd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.522698 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-systemd\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dd3e64a-9a74-4542-b09a-7003e749b44f-konnectivity-ca\") pod \"konnectivity-agent-zmcjg\" (UID: \"3dd3e64a-9a74-4542-b09a-7003e749b44f\") " pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-tuned\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522345 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-systemd\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-hostroot\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28de4e55-40b4-4721-9f39-2bbd1033c06e-host-slash\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522391 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522411 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-binary-copy\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-env-overrides\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522477 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac082af6-5ee0-4995-9636-8271debb950a-serviceca\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522483 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28de4e55-40b4-4721-9f39-2bbd1033c06e-host-slash\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522510 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-hostroot\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522528 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-sys-fs\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-node-log\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-tmp-dir\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac082af6-5ee0-4995-9636-8271debb950a-host\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:27:44.523513 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522638 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-sys-fs\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysconfig\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522660 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-node-log\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522627 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522705 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysconfig\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-conf-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522741 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac082af6-5ee0-4995-9636-8271debb950a-host\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckh87\" (UniqueName: \"kubernetes.io/projected/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-kube-api-access-ckh87\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-host\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522785 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-env-overrides\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-kubelet\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-system-cni-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522850 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-host\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522893 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-system-cni-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522894 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-kubelet\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dd3e64a-9a74-4542-b09a-7003e749b44f-agent-certs\") pod \"konnectivity-agent-zmcjg\" (UID: \"3dd3e64a-9a74-4542-b09a-7003e749b44f\") " pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522926 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-conf-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522955 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-tmp-dir\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:27:44.524337 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.522959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-slash\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523014 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-slash\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-etc-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523115 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-etc-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-socket-dir-parent\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-multus-certs\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-netns\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq8vh\" (UniqueName: \"kubernetes.io/projected/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-kube-api-access-dq8vh\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-system-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-k8s-cni-cncf-io\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-multus\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523417 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523437 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-k8s-cni-cncf-io\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523467 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-multus-certs\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523477 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-socket-dir-parent\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523514 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-socket-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-registration-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.524988 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcs67\" (UniqueName: \"kubernetes.io/projected/288945de-f0e7-47c7-87c9-45f8dbc021d9-kube-api-access-xcs67\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523647 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-system-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-sys\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-bin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-etc-kubernetes\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cnibin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmkwg\" (UniqueName: \"kubernetes.io/projected/fb187718-37b3-47fa-8c44-cd650c0185be-kube-api-access-nmkwg\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-kubelet-config\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-ovn\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523837 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-lib-modules\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523870 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-sys\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.525464 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-config\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.523944 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524017 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-ovn\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524103 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-socket-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524169 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-multus\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524205 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524250 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-registration-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524296 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cnibin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524330 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-bin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524331 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-etc-kubernetes\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.524427 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524435 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-lib-modules\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524447 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-netns\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.524474 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs podName:ffc3ecb5-d1e9-4125-a700-7c4b992b75bf nodeName:}" failed. No retries permitted until 2026-10-31 03:27:45.024459858 +0000 UTC m=+2.331576885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs") pod "network-metrics-daemon-rh4rl" (UID: "ffc3ecb5-d1e9-4125-a700-7c4b992b75bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.524512 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-kubelet-config\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.525624 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-tuned\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.526092 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.525667 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-tmp\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.526570 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.525780 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovn-node-metrics-cert\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.526570 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.525888 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dd3e64a-9a74-4542-b09a-7003e749b44f-konnectivity-ca\") pod \"konnectivity-agent-zmcjg\" (UID: \"3dd3e64a-9a74-4542-b09a-7003e749b44f\") " pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:44.526570 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.525912 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-config\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.526570 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.525920 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.526570 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.525938 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.527418 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.527399 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dd3e64a-9a74-4542-b09a-7003e749b44f-agent-certs\") pod \"konnectivity-agent-zmcjg\" (UID: \"3dd3e64a-9a74-4542-b09a-7003e749b44f\") " pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:44.576487 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.576455 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:27:44.576487 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.576485 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:27:44.576667 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.576499 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p76gl for pod openshift-network-diagnostics/network-check-target-bh9xh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:27:44.576667 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:44.576558 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl podName:fc1bac22-a12b-4777-989e-a4c059633fd1 nodeName:}" failed. No retries permitted until 2026-10-31 03:27:45.076540434 +0000 UTC m=+2.383657466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p76gl" (UniqueName: "kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl") pod "network-check-target-bh9xh" (UID: "fc1bac22-a12b-4777-989e-a4c059633fd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:27:44.579295 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.578992 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgl9x\" (UniqueName: \"kubernetes.io/projected/ac082af6-5ee0-4995-9636-8271debb950a-kube-api-access-kgl9x\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:27:44.579295 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.579183 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpjn\" (UniqueName: \"kubernetes.io/projected/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kube-api-access-jkpjn\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.579295 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.579248 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqlc\" (UniqueName: \"kubernetes.io/projected/f7bf98c9-3458-4936-8dfd-32faa8e522b9-kube-api-access-mdqlc\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.579564 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.579305 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtjc\" (UniqueName: \"kubernetes.io/projected/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-kube-api-access-hrtjc\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.579564 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.579526 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckh87\" (UniqueName: \"kubernetes.io/projected/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-kube-api-access-ckh87\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:44.579962 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.579945 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdfsd\" (UniqueName: \"kubernetes.io/projected/28de4e55-40b4-4721-9f39-2bbd1033c06e-kube-api-access-zdfsd\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:27:44.580594 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.580545 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq8vh\" (UniqueName: \"kubernetes.io/projected/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-kube-api-access-dq8vh\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:27:44.580702 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.580688 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcs67\" (UniqueName: \"kubernetes.io/projected/288945de-f0e7-47c7-87c9-45f8dbc021d9-kube-api-access-xcs67\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.581198 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.581180 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmkwg\" (UniqueName: \"kubernetes.io/projected/fb187718-37b3-47fa-8c44-cd650c0185be-kube-api-access-nmkwg\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.657425 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.657395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2vww2" Oct 31 03:27:44.677840 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.677812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:27:44.705171 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.705146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:27:44.710033 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.710018 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:44.720089 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.720066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:27:44.722223 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.722198 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:27:44.739903 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.739880 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:27:44.778037 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.777995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9sx4r" Oct 31 03:27:44.783234 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.783209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:27:44.881740 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:44.881722 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:27:45.027588 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:45.027551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:45.027757 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:45.027593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:45.027757 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:45.027701 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:27:45.027830 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:45.027759 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs podName:ffc3ecb5-d1e9-4125-a700-7c4b992b75bf nodeName:}" failed. No retries permitted until 2026-10-31 03:27:46.027745739 +0000 UTC m=+3.334862756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs") pod "network-metrics-daemon-rh4rl" (UID: "ffc3ecb5-d1e9-4125-a700-7c4b992b75bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:27:45.027830 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:45.027704 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:27:45.027830 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:45.027798 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret podName:6e9ea6af-1755-4e15-861b-5740354972f2 nodeName:}" failed. No retries permitted until 2026-10-31 03:27:46.027791785 +0000 UTC m=+3.334908796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret") pod "global-pull-secret-syncer-n9rx6" (UID: "6e9ea6af-1755-4e15-861b-5740354972f2") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:27:45.128273 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:45.128193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76gl\" (UniqueName: \"kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl\") pod \"network-check-target-bh9xh\" (UID: \"fc1bac22-a12b-4777-989e-a4c059633fd1\") " pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:45.128767 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:45.128345 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:27:45.128767 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:45.128363 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:27:45.128767 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:45.128374 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p76gl for pod openshift-network-diagnostics/network-check-target-bh9xh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:27:45.128767 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:45.128419 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl podName:fc1bac22-a12b-4777-989e-a4c059633fd1 nodeName:}" failed. No retries permitted until 2026-10-31 03:27:46.128406161 +0000 UTC m=+3.435523172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p76gl" (UniqueName: "kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl") pod "network-check-target-bh9xh" (UID: "fc1bac22-a12b-4777-989e-a4c059633fd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:27:45.246527 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:45.246484 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" event={"ID":"2625f366498146fdc680add27e4d568b","Type":"ContainerStarted","Data":"95281851c410cc7594f00a32f18b184d78b032a186a5d16a25e5c0eb854a1004"} Oct 31 03:27:45.247561 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:45.247543 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" event={"ID":"506bf97362686c30bca7e209bcab60a8","Type":"ContainerStarted","Data":"43c43b2135c93e9451676dc9f787280ef50687f9576b4c7c534716851e166aa8"} Oct 31 03:27:45.437275 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:45.437221 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:22:44 +0000 UTC" deadline="2028-05-18 17:11:57.290147062 +0000 UTC" Oct 31 03:27:45.437275 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:45.437263 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13573h44m11.852886603s" Oct 31 03:27:45.441483 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:45.441464 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:27:45.546307 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:45.546275 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb187718_37b3_47fa_8c44_cd650c0185be.slice/crio-8d02727ef543b5eace85884e096e8f1f3500c67b05de6ef37074a7c06c9235a5 WatchSource:0}: Error finding container 8d02727ef543b5eace85884e096e8f1f3500c67b05de6ef37074a7c06c9235a5: Status 404 returned error can't find the container with id 8d02727ef543b5eace85884e096e8f1f3500c67b05de6ef37074a7c06c9235a5 Oct 31 03:27:45.565545 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:45.565516 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac082af6_5ee0_4995_9636_8271debb950a.slice/crio-cf51eab920a360395e0836861207f7b44d760a326ed20f7860e998a0cfd1befe WatchSource:0}: Error finding container cf51eab920a360395e0836861207f7b44d760a326ed20f7860e998a0cfd1befe: Status 404 returned error can't find the container with id cf51eab920a360395e0836861207f7b44d760a326ed20f7860e998a0cfd1befe Oct 31 03:27:45.581084 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:45.581061 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb1e064_f4aa_4576_ae1e_a7bc8ffa7d49.slice/crio-6a498898d9587d10dcdfacda5b01fe6b4bbb138bd8ba2c78e66c9e6b6bfcc8b0 WatchSource:0}: Error finding container 6a498898d9587d10dcdfacda5b01fe6b4bbb138bd8ba2c78e66c9e6b6bfcc8b0: Status 404 returned error can't find the container with id 6a498898d9587d10dcdfacda5b01fe6b4bbb138bd8ba2c78e66c9e6b6bfcc8b0 Oct 31 03:27:45.608911 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:45.608881 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c37d3a_09c1_48a2_bd7b_abf3975ca24e.slice/crio-dbb070557619b8b33857fdef7b5d6cd213390898069caf0044daf30a84a0ae94 WatchSource:0}: Error finding container dbb070557619b8b33857fdef7b5d6cd213390898069caf0044daf30a84a0ae94: Status 404 returned error can't find the container with id dbb070557619b8b33857fdef7b5d6cd213390898069caf0044daf30a84a0ae94 Oct 31 03:27:45.619355 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:45.619327 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bf98c9_3458_4936_8dfd_32faa8e522b9.slice/crio-93d47675f3b545e0333b1e58c69b3e39aa2a346b104e93372ad01a575de407ff WatchSource:0}: Error finding container 93d47675f3b545e0333b1e58c69b3e39aa2a346b104e93372ad01a575de407ff: Status 404 returned error can't find the container with id 93d47675f3b545e0333b1e58c69b3e39aa2a346b104e93372ad01a575de407ff Oct 31 03:27:45.640911 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:45.640886 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:27:45.659154 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:45.659115 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28de4e55_40b4_4721_9f39_2bbd1033c06e.slice/crio-5c323437e958654bbc17871bc5fbdbcbe6c0ccf05d539eb833181ddb4fecff4a WatchSource:0}: Error finding container 5c323437e958654bbc17871bc5fbdbcbe6c0ccf05d539eb833181ddb4fecff4a: Status 404 returned error can't find the container with id 5c323437e958654bbc17871bc5fbdbcbe6c0ccf05d539eb833181ddb4fecff4a Oct 31 03:27:45.659853 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:45.659839 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288945de_f0e7_47c7_87c9_45f8dbc021d9.slice/crio-32d421ad4cf8e978e969ee60350ce7c259cd11aab322e053b6e7920de73497c6 WatchSource:0}: Error finding container 32d421ad4cf8e978e969ee60350ce7c259cd11aab322e053b6e7920de73497c6: Status 404 returned error can't find the container with id 32d421ad4cf8e978e969ee60350ce7c259cd11aab322e053b6e7920de73497c6 Oct 31 03:27:45.749385 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:45.749353 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd3e64a_9a74_4542_b09a_7003e749b44f.slice/crio-7cca36f210d4a8551823b7e0b90e3d8160ce272085ac73c1722b039719568e6b WatchSource:0}: Error finding container 7cca36f210d4a8551823b7e0b90e3d8160ce272085ac73c1722b039719568e6b: Status 404 returned error can't find the container with id 7cca36f210d4a8551823b7e0b90e3d8160ce272085ac73c1722b039719568e6b Oct 31 03:27:45.750702 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:27:45.750671 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2dbb0c_6baf_43d3_8fa9_3534f9fe8eb0.slice/crio-00747817d41ac8c6c61135aa78011e89d041d1bf49759cfff7aaffda9f9c3be2 WatchSource:0}: Error finding container 00747817d41ac8c6c61135aa78011e89d041d1bf49759cfff7aaffda9f9c3be2: Status 404 returned error can't find the container with id 00747817d41ac8c6c61135aa78011e89d041d1bf49759cfff7aaffda9f9c3be2 Oct 31 03:27:46.033902 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.033812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:46.033902 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.033873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:46.034079 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:46.033985 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:27:46.034079 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:46.034051 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs podName:ffc3ecb5-d1e9-4125-a700-7c4b992b75bf nodeName:}" failed. No retries permitted until 2026-10-31 03:27:48.034037196 +0000 UTC m=+5.341154208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs") pod "network-metrics-daemon-rh4rl" (UID: "ffc3ecb5-d1e9-4125-a700-7c4b992b75bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:27:46.034079 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:46.033985 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:27:46.034196 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:46.034096 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret podName:6e9ea6af-1755-4e15-861b-5740354972f2 nodeName:}" failed. No retries permitted until 2026-10-31 03:27:48.03408741 +0000 UTC m=+5.341204426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret") pod "global-pull-secret-syncer-n9rx6" (UID: "6e9ea6af-1755-4e15-861b-5740354972f2") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:27:46.134647 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.134617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76gl\" (UniqueName: \"kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl\") pod \"network-check-target-bh9xh\" (UID: \"fc1bac22-a12b-4777-989e-a4c059633fd1\") " pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:46.135116 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:46.134767 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:27:46.135116 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:46.134791 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:27:46.135116 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:46.134802 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p76gl for pod openshift-network-diagnostics/network-check-target-bh9xh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:27:46.135116 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:46.134864 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl podName:fc1bac22-a12b-4777-989e-a4c059633fd1 nodeName:}" failed. No retries permitted until 2026-10-31 03:27:48.134846872 +0000 UTC m=+5.441963899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p76gl" (UniqueName: "kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl") pod "network-check-target-bh9xh" (UID: "fc1bac22-a12b-4777-989e-a4c059633fd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:27:46.242968 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.242708 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:46.242968 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.242786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:46.242968 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.242895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:46.242968 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:46.242912 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:27:46.242968 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:46.242970 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:27:46.243389 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:46.243059 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:27:46.250318 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.250284 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmcjg" event={"ID":"3dd3e64a-9a74-4542-b09a-7003e749b44f","Type":"ContainerStarted","Data":"7cca36f210d4a8551823b7e0b90e3d8160ce272085ac73c1722b039719568e6b"} Oct 31 03:27:46.251659 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.251632 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lt92b" event={"ID":"28de4e55-40b4-4721-9f39-2bbd1033c06e","Type":"ContainerStarted","Data":"5c323437e958654bbc17871bc5fbdbcbe6c0ccf05d539eb833181ddb4fecff4a"} Oct 31 03:27:46.252835 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.252808 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" event={"ID":"f7bf98c9-3458-4936-8dfd-32faa8e522b9","Type":"ContainerStarted","Data":"93d47675f3b545e0333b1e58c69b3e39aa2a346b104e93372ad01a575de407ff"} Oct 31 03:27:46.254278 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.254250 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" event={"ID":"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e","Type":"ContainerStarted","Data":"dbb070557619b8b33857fdef7b5d6cd213390898069caf0044daf30a84a0ae94"} Oct 31 03:27:46.255744 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.255690 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7txzf" event={"ID":"fb187718-37b3-47fa-8c44-cd650c0185be","Type":"ContainerStarted","Data":"8d02727ef543b5eace85884e096e8f1f3500c67b05de6ef37074a7c06c9235a5"} Oct 31 03:27:46.256933 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.256904 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9sx4r" event={"ID":"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0","Type":"ContainerStarted","Data":"00747817d41ac8c6c61135aa78011e89d041d1bf49759cfff7aaffda9f9c3be2"} Oct 31 03:27:46.258416 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.258385 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" event={"ID":"288945de-f0e7-47c7-87c9-45f8dbc021d9","Type":"ContainerStarted","Data":"32d421ad4cf8e978e969ee60350ce7c259cd11aab322e053b6e7920de73497c6"} Oct 31 03:27:46.259507 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.259486 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2vww2" event={"ID":"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49","Type":"ContainerStarted","Data":"6a498898d9587d10dcdfacda5b01fe6b4bbb138bd8ba2c78e66c9e6b6bfcc8b0"} Oct 31 03:27:46.260633 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.260609 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fb4pz" event={"ID":"ac082af6-5ee0-4995-9636-8271debb950a","Type":"ContainerStarted","Data":"cf51eab920a360395e0836861207f7b44d760a326ed20f7860e998a0cfd1befe"} Oct 31 03:27:46.438472 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.438408 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:22:44 +0000 UTC" deadline="2028-04-28 03:51:40.904814132 +0000 UTC" Oct 31 03:27:46.438472 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:46.438465 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13080h23m54.466353316s" Oct 31 03:27:48.050797 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:48.049988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:48.050797 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:48.050051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:48.050797 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:48.050218 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:27:48.050797 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:48.050287 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret podName:6e9ea6af-1755-4e15-861b-5740354972f2 nodeName:}" failed. No retries permitted until 2026-10-31 03:27:52.050268182 +0000 UTC m=+9.357385194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret") pod "global-pull-secret-syncer-n9rx6" (UID: "6e9ea6af-1755-4e15-861b-5740354972f2") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:27:48.050797 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:48.050711 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:27:48.050797 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:48.050769 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs podName:ffc3ecb5-d1e9-4125-a700-7c4b992b75bf nodeName:}" failed. No retries permitted until 2026-10-31 03:27:52.050750915 +0000 UTC m=+9.357867928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs") pod "network-metrics-daemon-rh4rl" (UID: "ffc3ecb5-d1e9-4125-a700-7c4b992b75bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:27:48.151300 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:48.151261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76gl\" (UniqueName: \"kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl\") pod \"network-check-target-bh9xh\" (UID: \"fc1bac22-a12b-4777-989e-a4c059633fd1\") " pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:48.151488 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:48.151446 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:27:48.151488 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:48.151466 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:27:48.151488 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:48.151480 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p76gl for pod openshift-network-diagnostics/network-check-target-bh9xh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:27:48.151655 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:48.151539 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl podName:fc1bac22-a12b-4777-989e-a4c059633fd1 nodeName:}" failed. No retries permitted until 2026-10-31 03:27:52.151521862 +0000 UTC m=+9.458638877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p76gl" (UniqueName: "kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl") pod "network-check-target-bh9xh" (UID: "fc1bac22-a12b-4777-989e-a4c059633fd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:27:48.242593 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:48.242557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:48.242772 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:48.242695 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:27:48.243114 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:48.243076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:48.243232 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:48.243201 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:27:48.243294 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:48.243249 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:48.243345 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:48.243314 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:27:50.243162 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:50.242918 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:50.243664 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:50.243253 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:27:50.243664 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:50.243329 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:50.243664 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:50.243393 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:27:50.243664 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:50.243450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:50.243664 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:50.243531 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:27:50.290660 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:50.290624 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="9ba8cbf7ee466215fb50fa1b0261e3e8fb5eb5e1f98acb467caecc348f9e66cc" exitCode=0 Oct 31 03:27:50.290865 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:50.290713 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7txzf" event={"ID":"fb187718-37b3-47fa-8c44-cd650c0185be","Type":"ContainerDied","Data":"9ba8cbf7ee466215fb50fa1b0261e3e8fb5eb5e1f98acb467caecc348f9e66cc"} Oct 31 03:27:50.296187 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:50.296162 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" event={"ID":"506bf97362686c30bca7e209bcab60a8","Type":"ContainerStarted","Data":"bc4df7d9d76057e118f61bd8a22bea793611057b24a01f8dc4842016f2de7d84"} Oct 31 03:27:50.303893 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:50.303865 2576 generic.go:356] "Generic (PLEG): container finished" podID="2625f366498146fdc680add27e4d568b" containerID="d9a9779dc13e6c7deae08f64daf66f6acda9514da28258e41a06865b4758ec05" exitCode=0 Oct 31 03:27:50.303984 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:50.303910 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" event={"ID":"2625f366498146fdc680add27e4d568b","Type":"ContainerDied","Data":"d9a9779dc13e6c7deae08f64daf66f6acda9514da28258e41a06865b4758ec05"} Oct 31 03:27:51.309822 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:51.309768 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" event={"ID":"2625f366498146fdc680add27e4d568b","Type":"ContainerStarted","Data":"9d10f036909470030826cbaaf856fb4d395479533d3159a0cd77881c9a6e4ea5"} Oct 31 03:27:51.314753 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:51.314720 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmcjg" event={"ID":"3dd3e64a-9a74-4542-b09a-7003e749b44f","Type":"ContainerStarted","Data":"9da1d7d6d9d32731b50f17126db297d1d8eb72dd3029ac00413492d5bea4319d"} Oct 31 03:27:51.428155 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:51.428068 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" podStartSLOduration=8.428043921 podStartE2EDuration="8.428043921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:50.743253498 +0000 UTC m=+8.050370560" watchObservedRunningTime="2026-10-31 03:27:51.428043921 +0000 UTC m=+8.735160956" Oct 31 03:27:51.545727 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:51.544926 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" podStartSLOduration=8.544904656 podStartE2EDuration="8.544904656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:51.428267615 +0000 UTC m=+8.735384669" watchObservedRunningTime="2026-10-31 03:27:51.544904656 +0000 UTC m=+8.852021692" Oct 31 03:27:52.092562 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:52.092522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:52.092748 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:52.092584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:52.092833 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:52.092745 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:27:52.092833 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:52.092809 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret podName:6e9ea6af-1755-4e15-861b-5740354972f2 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:00.092792041 +0000 UTC m=+17.399909090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret") pod "global-pull-secret-syncer-n9rx6" (UID: "6e9ea6af-1755-4e15-861b-5740354972f2") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:27:52.093270 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:52.093249 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:27:52.093341 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:52.093301 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs podName:ffc3ecb5-d1e9-4125-a700-7c4b992b75bf nodeName:}" failed. No retries permitted until 2026-10-31 03:28:00.093290522 +0000 UTC m=+17.400407547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs") pod "network-metrics-daemon-rh4rl" (UID: "ffc3ecb5-d1e9-4125-a700-7c4b992b75bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:27:52.193442 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:52.193404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76gl\" (UniqueName: \"kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl\") pod \"network-check-target-bh9xh\" (UID: \"fc1bac22-a12b-4777-989e-a4c059633fd1\") " pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:52.193622 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:52.193574 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:27:52.193622 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:52.193600 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:27:52.193622 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:52.193615 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p76gl for pod openshift-network-diagnostics/network-check-target-bh9xh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:27:52.193871 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:52.193673 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl podName:fc1bac22-a12b-4777-989e-a4c059633fd1 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:00.193653899 +0000 UTC m=+17.500770938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p76gl" (UniqueName: "kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl") pod "network-check-target-bh9xh" (UID: "fc1bac22-a12b-4777-989e-a4c059633fd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:27:52.242625 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:52.242583 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:52.242792 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:52.242728 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:27:52.242839 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:52.242804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:52.242896 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:52.242872 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:27:52.242964 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:52.242924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:52.243014 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:52.242997 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:27:52.324618 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:52.324092 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fb4pz" event={"ID":"ac082af6-5ee0-4995-9636-8271debb950a","Type":"ContainerStarted","Data":"2eb29065cf0a1f302a8945ff93384b1faf397fbb18676d3c8742d745201ffcac"} Oct 31 03:27:52.444173 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:52.440344 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zmcjg" podStartSLOduration=4.756906768 podStartE2EDuration="9.440324111s" totalImagesPullingTime="4.683417343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:51.545687149 +0000 UTC m=+8.852804188" watchObservedRunningTime="2026-10-31 03:27:52.440324111 +0000 UTC m=+9.747441146" Oct 31 03:27:53.332912 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:53.332870 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" event={"ID":"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e","Type":"ContainerStarted","Data":"a2dcc97299cd42dec52f84178b0081fa4f897b0ce3628ebed8e535b52c17f611"} Oct 31 03:27:54.243108 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:54.243069 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:54.243287 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:54.243234 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:27:54.243370 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:54.243329 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:54.243464 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:54.243419 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:27:54.243534 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:54.243479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:54.243593 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:54.243544 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:27:54.339400 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:54.339315 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9sx4r" event={"ID":"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0","Type":"ContainerStarted","Data":"87767443f9db8261d39042536603e53d0fe0895f71c41548f45a0307b902222c"} Oct 31 03:27:54.502667 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:54.502606 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fb4pz" podStartSLOduration=5.607246285 podStartE2EDuration="11.502587417s" totalImagesPullingTime="5.895341132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:52.448350681 +0000 UTC m=+9.755467716" watchObservedRunningTime="2026-10-31 03:27:54.502587417 +0000 UTC m=+11.809704455" Oct 31 03:27:54.741083 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:54.740849 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:27:55.316405 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:55.315572 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:27:54.741056985Z","UUID":"7e8976ea-2e0b-4bc1-a801-7f018f25df5f","Handler":null,"Name":"","Endpoint":""} Oct 31 03:27:55.317342 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:55.317316 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 03:27:55.317466 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:55.317348 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:27:55.344246 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:55.343785 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2vww2" event={"ID":"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49","Type":"ContainerStarted","Data":"181d34e82a90e6606f951b81656e955626e2cba82f665cc9eec131d2074d8be7"} Oct 31 03:27:55.346589 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:55.346559 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" event={"ID":"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e","Type":"ContainerStarted","Data":"56575d4c3f99f190c018dc1e6a8a5b24e2e82b1f68ce18066bb8e7f239d7fc89"} Oct 31 03:27:55.348754 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:55.348720 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="ac35a60f58205cce2789cef0093b27b2bd6fce80cdf95d137ed8ce1fe4062b19" exitCode=0 Oct 31 03:27:55.348894 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:55.348779 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7txzf" event={"ID":"fb187718-37b3-47fa-8c44-cd650c0185be","Type":"ContainerDied","Data":"ac35a60f58205cce2789cef0093b27b2bd6fce80cdf95d137ed8ce1fe4062b19"} Oct 31 03:27:55.461080 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:55.460185 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9sx4r" podStartSLOduration=4.13120904 podStartE2EDuration="12.460167557s" totalImagesPullingTime="8.328958517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:54.503407565 +0000 UTC m=+11.810524600" watchObservedRunningTime="2026-10-31 03:27:55.460167557 +0000 UTC m=+12.767284589" Oct 31 03:27:55.461080 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:55.460404 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2vww2" podStartSLOduration=3.781937377 podStartE2EDuration="12.460395709s" totalImagesPullingTime="8.678458332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:55.460135957 +0000 UTC m=+12.767253015" watchObservedRunningTime="2026-10-31 03:27:55.460395709 +0000 UTC m=+12.767512743" Oct 31 03:27:55.805018 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:55.804986 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:55.805767 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:55.805745 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:56.243557 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:56.243476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:56.243557 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:56.243496 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:56.243812 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:56.243484 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:56.243812 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:56.243606 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:27:56.243812 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:56.243685 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:27:56.244026 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:56.243805 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:27:56.352582 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:56.352515 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lt92b" event={"ID":"28de4e55-40b4-4721-9f39-2bbd1033c06e","Type":"ContainerStarted","Data":"c24251957093d2d5280158ab538f2551090d995d530f424bee3434a13f5f72fe"} Oct 31 03:27:56.353149 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:56.352779 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:56.353557 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:56.353511 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:27:56.469588 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:56.469509 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lt92b" podStartSLOduration=4.647233559 podStartE2EDuration="13.46948957s" totalImagesPullingTime="8.822256011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:56.46941899 +0000 UTC m=+13.776536024" watchObservedRunningTime="2026-10-31 03:27:56.46948957 +0000 UTC m=+13.776606605" Oct 31 03:27:57.356097 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:57.356056 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" event={"ID":"f7bf98c9-3458-4936-8dfd-32faa8e522b9","Type":"ContainerStarted","Data":"679d74c26ef05c9daf82f220ad1161dfaf8a154b3653c314900b8cffe5da95c4"} Oct 31 03:27:57.358612 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:57.358587 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" event={"ID":"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e","Type":"ContainerStarted","Data":"318338a86799cee84bef1b1d5fcc9573e1e6b8a9d24444c70a88bb410b9b77c8"} Oct 31 03:27:57.360686 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:57.360658 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="07e5d03f4e2835eb6f14f7dd49454fae88f00c8723e9104604484512b1d277c3" exitCode=0 Oct 31 03:27:57.360805 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:57.360763 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7txzf" event={"ID":"fb187718-37b3-47fa-8c44-cd650c0185be","Type":"ContainerDied","Data":"07e5d03f4e2835eb6f14f7dd49454fae88f00c8723e9104604484512b1d277c3"} Oct 31 03:27:57.477091 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:57.477037 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" podStartSLOduration=3.652099221 podStartE2EDuration="14.477020703s" totalImagesPullingTime="10.824921482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:57.476960911 +0000 UTC m=+14.784077993" watchObservedRunningTime="2026-10-31 03:27:57.477020703 +0000 UTC m=+14.784137742" Oct 31 03:27:57.730387 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:57.730331 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" podStartSLOduration=4.493121422 podStartE2EDuration="14.730311427s" totalImagesPullingTime="10.237190005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:57.729379825 +0000 UTC m=+15.036496861" watchObservedRunningTime="2026-10-31 03:27:57.730311427 +0000 UTC m=+15.037428462" Oct 31 03:27:58.242893 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:58.242859 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:27:58.242893 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:58.242891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:27:58.243166 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:58.242984 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:27:58.243166 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:58.243045 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:27:58.243166 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:58.243087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:27:58.243296 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:27:58.243193 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:27:59.368495 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:59.368284 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="e67324b77856497ea641ddda29022dd487b25f71c54d0f6b5ebd361df9b113fc" exitCode=0 Oct 31 03:27:59.368495 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:27:59.368381 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7txzf" event={"ID":"fb187718-37b3-47fa-8c44-cd650c0185be","Type":"ContainerDied","Data":"e67324b77856497ea641ddda29022dd487b25f71c54d0f6b5ebd361df9b113fc"} Oct 31 03:28:00.158240 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:00.158206 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:00.158438 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:00.158255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:00.158438 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:00.158381 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:28:00.158438 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:00.158387 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:28:00.158438 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:00.158430 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret podName:6e9ea6af-1755-4e15-861b-5740354972f2 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:16.158418259 +0000 UTC m=+33.465535272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret") pod "global-pull-secret-syncer-n9rx6" (UID: "6e9ea6af-1755-4e15-861b-5740354972f2") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:28:00.158578 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:00.158445 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs podName:ffc3ecb5-d1e9-4125-a700-7c4b992b75bf nodeName:}" failed. No retries permitted until 2026-10-31 03:28:16.158438271 +0000 UTC m=+33.465555282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs") pod "network-metrics-daemon-rh4rl" (UID: "ffc3ecb5-d1e9-4125-a700-7c4b992b75bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:28:00.243014 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:00.242977 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:00.243189 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:00.243019 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:00.243189 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:00.243019 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:00.243189 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:00.243102 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:28:00.243353 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:00.243204 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:28:00.243353 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:00.243297 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:28:00.259215 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:00.259177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76gl\" (UniqueName: \"kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl\") pod \"network-check-target-bh9xh\" (UID: \"fc1bac22-a12b-4777-989e-a4c059633fd1\") " pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:00.259367 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:00.259347 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:28:00.259436 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:00.259369 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:28:00.259436 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:00.259379 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p76gl for pod openshift-network-diagnostics/network-check-target-bh9xh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:28:00.259436 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:00.259423 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl podName:fc1bac22-a12b-4777-989e-a4c059633fd1 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:16.259409136 +0000 UTC m=+33.566526148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p76gl" (UniqueName: "kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl") pod "network-check-target-bh9xh" (UID: "fc1bac22-a12b-4777-989e-a4c059633fd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:28:01.376502 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:01.376234 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" event={"ID":"288945de-f0e7-47c7-87c9-45f8dbc021d9","Type":"ContainerStarted","Data":"8ea118ef49d6c0dfc994f00b954fb5439bd0bc9033f74792326151a615c9987c"} Oct 31 03:28:01.376922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:01.376514 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" event={"ID":"288945de-f0e7-47c7-87c9-45f8dbc021d9","Type":"ContainerStarted","Data":"ae1062423775d010b90240b4d15b3434e3d3ea5d1aabc8c5964bac86c4f58a20"} Oct 31 03:28:01.376922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:01.376572 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" event={"ID":"288945de-f0e7-47c7-87c9-45f8dbc021d9","Type":"ContainerStarted","Data":"8acf33774c7bd4aa06284306a692f0a1a1ed96c7b5ea28a4f0bc6b85db54c75f"} Oct 31 03:28:01.376922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:01.376620 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" event={"ID":"288945de-f0e7-47c7-87c9-45f8dbc021d9","Type":"ContainerStarted","Data":"9ef0ffd57fff8b26b3126c8cd0bdaf132ea52539d48aaa1bd056a864e6ba0c3e"} Oct 31 03:28:01.376922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:01.376667 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" event={"ID":"288945de-f0e7-47c7-87c9-45f8dbc021d9","Type":"ContainerStarted","Data":"2e35f62e89309fb1a061675d3cf5eac37f9ee33bfe21e8b6dfe94e449cb07498"} Oct 31 03:28:01.376922 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:01.376714 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" event={"ID":"288945de-f0e7-47c7-87c9-45f8dbc021d9","Type":"ContainerStarted","Data":"51c25fd54f520b8071e6ceab71491ff0b7de193c8a70a118ada32154411a7c39"} Oct 31 03:28:02.243383 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:02.243347 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:02.243522 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:02.243351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:02.243522 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:02.243469 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:28:02.243623 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:02.243351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:02.243623 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:02.243541 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:28:02.243623 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:02.243611 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:28:02.380947 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:02.380857 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="221b09445b5de3846ca20cc8ea5a6c5af83aded84e60276880c7079cff5fb61a" exitCode=0 Oct 31 03:28:02.380947 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:02.380893 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7txzf" event={"ID":"fb187718-37b3-47fa-8c44-cd650c0185be","Type":"ContainerDied","Data":"221b09445b5de3846ca20cc8ea5a6c5af83aded84e60276880c7079cff5fb61a"} Oct 31 03:28:03.386702 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:03.386667 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="80cafb369a21f6579d3e631174bdb2f0cc91b307d1e87076f2dc8847212a25ad" exitCode=0 Oct 31 03:28:03.387235 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:03.386758 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7txzf" event={"ID":"fb187718-37b3-47fa-8c44-cd650c0185be","Type":"ContainerDied","Data":"80cafb369a21f6579d3e631174bdb2f0cc91b307d1e87076f2dc8847212a25ad"} Oct 31 03:28:03.390559 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:03.390529 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" event={"ID":"288945de-f0e7-47c7-87c9-45f8dbc021d9","Type":"ContainerStarted","Data":"94b19ed50b9c448907cc863e4e37f38f4e5434872b7d1998fdb2ad20200914a0"} Oct 31 03:28:04.243501 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:04.243459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:04.243670 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:04.243545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:04.243708 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:04.243678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:04.243741 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:04.243692 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:28:04.243771 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:04.243746 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:28:04.243849 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:04.243823 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:28:04.395769 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:04.395731 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7txzf" event={"ID":"fb187718-37b3-47fa-8c44-cd650c0185be","Type":"ContainerStarted","Data":"4ab7af6d0020e2f47a12819f194ac03ac8cbd7b9aac88e8dee1914b8dab2c054"} Oct 31 03:28:04.522538 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:04.522413 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7txzf" podStartSLOduration=8.056781006 podStartE2EDuration="21.52239787s" totalImagesPullingTime="13.465616864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:04.519203346 +0000 UTC m=+21.826320379" watchObservedRunningTime="2026-10-31 03:28:04.52239787 +0000 UTC m=+21.829514953" Oct 31 03:28:06.242822 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:06.242638 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:06.243480 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:06.242645 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:06.243480 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:06.242899 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:28:06.243480 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:06.242696 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:06.243480 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:06.242964 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:28:06.243480 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:06.243059 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:28:06.407279 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:06.407248 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" event={"ID":"288945de-f0e7-47c7-87c9-45f8dbc021d9","Type":"ContainerStarted","Data":"5c5a46c4dfa09da15b38e6b526f9215533d739b99285c4ba2f17671862dc4a55"} Oct 31 03:28:06.407543 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:06.407513 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:06.407663 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:06.407582 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:06.422743 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:06.422722 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:06.690491 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:06.690425 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" podStartSLOduration=8.599874668 podStartE2EDuration="23.690403427s" totalImagesPullingTime="15.090528759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:06.638676386 +0000 UTC m=+23.945793420" watchObservedRunningTime="2026-10-31 03:28:06.690403427 +0000 UTC m=+23.997520463" Oct 31 03:28:07.412289 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:07.412260 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:07.427815 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:07.427790 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:08.093506 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:08.093419 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n9rx6"] Oct 31 03:28:08.093693 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:08.093548 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:08.093693 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:08.093636 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:28:08.094045 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:08.094024 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rh4rl"] Oct 31 03:28:08.094178 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:08.094164 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:08.094319 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:08.094299 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:28:08.095472 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:08.095451 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bh9xh"] Oct 31 03:28:08.095547 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:08.095534 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:08.095619 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:08.095603 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:28:09.243413 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:09.243376 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:09.243845 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:09.243530 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:28:10.242788 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:10.242744 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:10.242788 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:10.242795 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:10.242996 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:10.242876 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:28:10.243040 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:10.242995 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:28:11.242943 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:11.242904 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:11.243349 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:11.243051 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:28:12.242494 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:12.242446 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:12.242673 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:12.242499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:12.242673 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:12.242559 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9rx6" podUID="6e9ea6af-1755-4e15-861b-5740354972f2" Oct 31 03:28:12.242673 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:12.242663 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh9xh" podUID="fc1bac22-a12b-4777-989e-a4c059633fd1" Oct 31 03:28:13.243352 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:13.243190 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:13.243732 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:13.243424 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4rl" podUID="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" Oct 31 03:28:14.031950 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.031872 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeReady" Oct 31 03:28:14.032087 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.032015 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 03:28:14.187978 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.187939 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5rfb8"] Oct 31 03:28:14.206882 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.206854 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pr54w"] Oct 31 03:28:14.207047 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.206986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.220767 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.220744 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5rfb8"] Oct 31 03:28:14.220859 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.220781 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pr54w"] Oct 31 03:28:14.220859 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.220846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:14.242762 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.242738 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:14.242870 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.242740 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:14.260187 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.260166 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 03:28:14.260490 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.260350 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 03:28:14.260490 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.260387 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fs258\"" Oct 31 03:28:14.273852 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.273838 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 03:28:14.273994 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.273981 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 03:28:14.274041 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.274024 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 03:28:14.274074 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.274062 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dvnpq\"" Oct 31 03:28:14.356308 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.356218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.356308 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.356261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6qd\" (UniqueName: \"kubernetes.io/projected/ce341b3b-e888-41f3-90b2-2055a4634a2d-kube-api-access-rb6qd\") pod \"ingress-canary-pr54w\" (UID: \"ce341b3b-e888-41f3-90b2-2055a4634a2d\") " pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:14.356308 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.356289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4jzx\" (UniqueName: \"kubernetes.io/projected/ad4db9a5-414b-4e4c-a712-e30ae75254c6-kube-api-access-w4jzx\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.356574 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.356365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4db9a5-414b-4e4c-a712-e30ae75254c6-tmp-dir\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.356574 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.356434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce341b3b-e888-41f3-90b2-2055a4634a2d-cert\") pod \"ingress-canary-pr54w\" (UID: \"ce341b3b-e888-41f3-90b2-2055a4634a2d\") " pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:14.356574 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.356480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4db9a5-414b-4e4c-a712-e30ae75254c6-config-volume\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.360731 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.360714 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:28:14.360834 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.360776 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:28:14.360956 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.360943 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lj989\"" Oct 31 03:28:14.360992 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.360977 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-8f2x4\"" Oct 31 03:28:14.361027 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.361016 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:28:14.456999 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.456969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4db9a5-414b-4e4c-a712-e30ae75254c6-config-volume\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.457175 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.457022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.457175 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.457046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6qd\" (UniqueName: \"kubernetes.io/projected/ce341b3b-e888-41f3-90b2-2055a4634a2d-kube-api-access-rb6qd\") pod \"ingress-canary-pr54w\" (UID: \"ce341b3b-e888-41f3-90b2-2055a4634a2d\") " pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:14.457175 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.457084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4jzx\" (UniqueName: \"kubernetes.io/projected/ad4db9a5-414b-4e4c-a712-e30ae75254c6-kube-api-access-w4jzx\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.457175 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.457110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4db9a5-414b-4e4c-a712-e30ae75254c6-tmp-dir\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.457175 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:14.457152 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:28:14.457175 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.457158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce341b3b-e888-41f3-90b2-2055a4634a2d-cert\") pod \"ingress-canary-pr54w\" (UID: \"ce341b3b-e888-41f3-90b2-2055a4634a2d\") " pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:14.457435 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:14.457241 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls podName:ad4db9a5-414b-4e4c-a712-e30ae75254c6 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:14.957222022 +0000 UTC m=+32.264339057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls") pod "dns-default-5rfb8" (UID: "ad4db9a5-414b-4e4c-a712-e30ae75254c6") : secret "dns-default-metrics-tls" not found Oct 31 03:28:14.457504 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.457479 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4db9a5-414b-4e4c-a712-e30ae75254c6-tmp-dir\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.457669 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.457653 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4db9a5-414b-4e4c-a712-e30ae75254c6-config-volume\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.460911 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.460889 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce341b3b-e888-41f3-90b2-2055a4634a2d-cert\") pod \"ingress-canary-pr54w\" (UID: \"ce341b3b-e888-41f3-90b2-2055a4634a2d\") " pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:14.519575 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.519547 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6qd\" (UniqueName: \"kubernetes.io/projected/ce341b3b-e888-41f3-90b2-2055a4634a2d-kube-api-access-rb6qd\") pod \"ingress-canary-pr54w\" (UID: \"ce341b3b-e888-41f3-90b2-2055a4634a2d\") " pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:14.519717 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.519595 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4jzx\" (UniqueName: \"kubernetes.io/projected/ad4db9a5-414b-4e4c-a712-e30ae75254c6-kube-api-access-w4jzx\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.557071 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.557043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:14.960894 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:14.960813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:14.961034 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:14.960930 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:28:14.961034 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:14.960981 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls podName:ad4db9a5-414b-4e4c-a712-e30ae75254c6 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:15.960968632 +0000 UTC m=+33.268085658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls") pod "dns-default-5rfb8" (UID: "ad4db9a5-414b-4e4c-a712-e30ae75254c6") : secret "dns-default-metrics-tls" not found Oct 31 03:28:15.005430 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:28:15.005401 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce341b3b_e888_41f3_90b2_2055a4634a2d.slice/crio-e09b04ce16de788749087c777bf051dc1de26c2b669f1ba82f56e189cc523a82 WatchSource:0}: Error finding container e09b04ce16de788749087c777bf051dc1de26c2b669f1ba82f56e189cc523a82: Status 404 returned error can't find the container with id e09b04ce16de788749087c777bf051dc1de26c2b669f1ba82f56e189cc523a82 Oct 31 03:28:15.005807 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:15.005781 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pr54w"] Oct 31 03:28:15.243296 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:15.243218 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:15.296896 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:15.296865 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:28:15.296896 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:15.296875 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvwkt\"" Oct 31 03:28:15.427478 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:15.427441 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pr54w" event={"ID":"ce341b3b-e888-41f3-90b2-2055a4634a2d","Type":"ContainerStarted","Data":"e09b04ce16de788749087c777bf051dc1de26c2b669f1ba82f56e189cc523a82"} Oct 31 03:28:15.968414 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:15.968371 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:15.968579 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:15.968510 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:28:15.968579 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:15.968571 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls podName:ad4db9a5-414b-4e4c-a712-e30ae75254c6 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:17.968556432 +0000 UTC m=+35.275673487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls") pod "dns-default-5rfb8" (UID: "ad4db9a5-414b-4e4c-a712-e30ae75254c6") : secret "dns-default-metrics-tls" not found Oct 31 03:28:16.169790 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:16.169745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:16.169976 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:16.169958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:16.170082 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:16.170067 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 03:28:16.170166 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:16.170153 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs podName:ffc3ecb5-d1e9-4125-a700-7c4b992b75bf nodeName:}" failed. No retries permitted until 2026-10-31 03:28:48.170116928 +0000 UTC m=+65.477233966 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs") pod "network-metrics-daemon-rh4rl" (UID: "ffc3ecb5-d1e9-4125-a700-7c4b992b75bf") : secret "metrics-daemon-secret" not found Oct 31 03:28:16.172434 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:16.172405 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:16.270959 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:16.270887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76gl\" (UniqueName: \"kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl\") pod \"network-check-target-bh9xh\" (UID: \"fc1bac22-a12b-4777-989e-a4c059633fd1\") " pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:16.273652 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:16.273621 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76gl\" (UniqueName: \"kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl\") pod \"network-check-target-bh9xh\" (UID: \"fc1bac22-a12b-4777-989e-a4c059633fd1\") " pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:16.371308 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:16.371275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:16.386297 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:16.386266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:16.763276 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:16.763244 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n9rx6"] Oct 31 03:28:16.797405 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:28:16.797368 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9ea6af_1755_4e15_861b_5740354972f2.slice/crio-b3244e8d47504d7f3c3dc14d767806c746ec460ad7e15abef23e3e5be7d528d8 WatchSource:0}: Error finding container b3244e8d47504d7f3c3dc14d767806c746ec460ad7e15abef23e3e5be7d528d8: Status 404 returned error can't find the container with id b3244e8d47504d7f3c3dc14d767806c746ec460ad7e15abef23e3e5be7d528d8 Oct 31 03:28:16.815103 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:16.812703 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bh9xh"] Oct 31 03:28:17.434345 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:17.434307 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n9rx6" event={"ID":"6e9ea6af-1755-4e15-861b-5740354972f2","Type":"ContainerStarted","Data":"b3244e8d47504d7f3c3dc14d767806c746ec460ad7e15abef23e3e5be7d528d8"} Oct 31 03:28:17.435703 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:17.435673 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bh9xh" event={"ID":"fc1bac22-a12b-4777-989e-a4c059633fd1","Type":"ContainerStarted","Data":"39a17dd3b5102d0eb14f9984185641c1a39b5199b4f11f9f42f42c8521408719"} Oct 31 03:28:17.983573 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:17.983538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:17.983820 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:17.983703 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:28:17.983820 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:17.983796 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls podName:ad4db9a5-414b-4e4c-a712-e30ae75254c6 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:21.983774124 +0000 UTC m=+39.290891138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls") pod "dns-default-5rfb8" (UID: "ad4db9a5-414b-4e4c-a712-e30ae75254c6") : secret "dns-default-metrics-tls" not found Oct 31 03:28:19.443074 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:19.443031 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pr54w" event={"ID":"ce341b3b-e888-41f3-90b2-2055a4634a2d","Type":"ContainerStarted","Data":"0e38602f51b819f031c330215f595ca16e02b31372894137fac4f8a3e98a8f95"} Oct 31 03:28:19.564116 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:19.564028 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pr54w" podStartSLOduration=1.85862632 podStartE2EDuration="5.564009258s" totalImagesPullingTime="3.705382938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:19.563248702 +0000 UTC m=+36.870365765" watchObservedRunningTime="2026-10-31 03:28:19.564009258 +0000 UTC m=+36.871126291" Oct 31 03:28:20.430109 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.430070 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lhsn4"] Oct 31 03:28:20.470184 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.470144 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.489436 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.489403 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lhsn4"] Oct 31 03:28:20.603178 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.602750 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 03:28:20.603178 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.602811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.603178 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.602835 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 03:28:20.603178 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.602832 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 03:28:20.603178 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.602865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-crio-socket\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.603178 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.602883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-data-volume\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.603178 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.602899 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.603178 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.602919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz6fg\" (UniqueName: \"kubernetes.io/projected/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-api-access-gz6fg\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.603178 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.602949 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 03:28:20.603178 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.603078 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jdz67\"" Oct 31 03:28:20.691388 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.691306 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hnt22-469x9"] Oct 31 03:28:20.703736 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.703696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-crio-socket\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.703736 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.703747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-data-volume\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.703945 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.703770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.703945 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.703795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6fg\" (UniqueName: \"kubernetes.io/projected/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-api-access-gz6fg\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.703945 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.703872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.703945 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:20.703910 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:28:20.704084 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.703952 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-crio-socket\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.704084 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:20.703982 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:21.203964629 +0000 UTC m=+38.511081658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : configmap "kube-rbac-proxy" not found Oct 31 03:28:20.704084 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:20.703982 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:28:20.704084 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:20.704016 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:21.204009468 +0000 UTC m=+38.511126483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : secret "insights-runtime-extractor-tls" not found Oct 31 03:28:20.704262 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.704080 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-data-volume\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.709463 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.709443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.752212 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.752174 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hnt22-469x9"] Oct 31 03:28:20.762891 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.762861 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 03:28:20.762891 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.762885 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bzhcg\"" Oct 31 03:28:20.804586 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.804548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-trusted-ca-bundle\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.804772 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.804599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dkdj\" (UniqueName: \"kubernetes.io/projected/2d2a2662-e327-4427-a3e8-482de3a8ed26-kube-api-access-4dkdj\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.804772 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.804718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-service-ca-bundle\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.804888 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.804838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2a2662-e327-4427-a3e8-482de3a8ed26-archives-path\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.854045 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.854014 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 03:28:20.855865 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.855834 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6fg\" (UniqueName: \"kubernetes.io/projected/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-api-access-gz6fg\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:20.906263 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.906226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2a2662-e327-4427-a3e8-482de3a8ed26-archives-path\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.906431 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.906300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-trusted-ca-bundle\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.906431 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.906330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dkdj\" (UniqueName: \"kubernetes.io/projected/2d2a2662-e327-4427-a3e8-482de3a8ed26-kube-api-access-4dkdj\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.906431 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.906386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-service-ca-bundle\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.906718 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.906682 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2a2662-e327-4427-a3e8-482de3a8ed26-archives-path\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.906972 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.906941 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-service-ca-bundle\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.907221 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.907196 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-trusted-ca-bundle\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:20.969231 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:20.969148 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dkdj\" (UniqueName: \"kubernetes.io/projected/2d2a2662-e327-4427-a3e8-482de3a8ed26-kube-api-access-4dkdj\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:21.038202 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.038167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:21.208835 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.208801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:21.209003 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.208864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:21.209003 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:21.208978 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:28:21.209003 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:21.208992 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:28:21.209145 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:21.209055 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:22.209036837 +0000 UTC m=+39.516153852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : configmap "kube-rbac-proxy" not found Oct 31 03:28:21.209145 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:21.209073 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:22.209064592 +0000 UTC m=+39.516181672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : secret "insights-runtime-extractor-tls" not found Oct 31 03:28:21.450719 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.450682 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bh9xh" event={"ID":"fc1bac22-a12b-4777-989e-a4c059633fd1","Type":"ContainerStarted","Data":"f826db4699c40c6fdeaae42adfe919280583bb62a36a913647489259f9e7e278"} Oct 31 03:28:21.450953 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.450925 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:21.482815 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.482791 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hnt22-469x9"] Oct 31 03:28:21.483207 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:28:21.482925 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2a2662_e327_4427_a3e8_482de3a8ed26.slice/crio-093dbcd91688b4f46084730b8bf5451b0e6af412eaf61f864416411886017e0e WatchSource:0}: Error finding container 093dbcd91688b4f46084730b8bf5451b0e6af412eaf61f864416411886017e0e: Status 404 returned error can't find the container with id 093dbcd91688b4f46084730b8bf5451b0e6af412eaf61f864416411886017e0e Oct 31 03:28:21.570570 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.570512 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bh9xh" podStartSLOduration=34.25688178 podStartE2EDuration="38.57049766s" totalImagesPullingTime="4.31361588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:21.570438697 +0000 UTC m=+38.877555731" watchObservedRunningTime="2026-10-31 03:28:21.57049766 +0000 UTC m=+38.877614691" Oct 31 03:28:21.725656 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.725617 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq"] Oct 31 03:28:21.745245 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.745171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq" Oct 31 03:28:21.783921 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.783894 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq"] Oct 31 03:28:21.798658 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.798603 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q48mz\"" Oct 31 03:28:21.798658 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.798606 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 03:28:21.798658 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.798635 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 03:28:21.913936 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:21.913884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5hs6\" (UniqueName: \"kubernetes.io/projected/14d2a2ab-d78a-47c1-8e0c-84f7418d2b81-kube-api-access-c5hs6\") pod \"migrator-66df5d496b-grlcq\" (UID: \"14d2a2ab-d78a-47c1-8e0c-84f7418d2b81\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq" Oct 31 03:28:22.015582 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.015161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:22.015582 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.015216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5hs6\" (UniqueName: \"kubernetes.io/projected/14d2a2ab-d78a-47c1-8e0c-84f7418d2b81-kube-api-access-c5hs6\") pod \"migrator-66df5d496b-grlcq\" (UID: \"14d2a2ab-d78a-47c1-8e0c-84f7418d2b81\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq" Oct 31 03:28:22.015582 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:22.015343 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:28:22.015582 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:22.015421 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls podName:ad4db9a5-414b-4e4c-a712-e30ae75254c6 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:30.015400137 +0000 UTC m=+47.322517165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls") pod "dns-default-5rfb8" (UID: "ad4db9a5-414b-4e4c-a712-e30ae75254c6") : secret "dns-default-metrics-tls" not found Oct 31 03:28:22.077097 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.077068 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5hs6\" (UniqueName: \"kubernetes.io/projected/14d2a2ab-d78a-47c1-8e0c-84f7418d2b81-kube-api-access-c5hs6\") pod \"migrator-66df5d496b-grlcq\" (UID: \"14d2a2ab-d78a-47c1-8e0c-84f7418d2b81\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq" Oct 31 03:28:22.216699 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.216656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:22.216853 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.216711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:22.216853 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:22.216806 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:28:22.216853 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:22.216816 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:28:22.216945 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:22.216873 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:24.216858528 +0000 UTC m=+41.523975544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : secret "insights-runtime-extractor-tls" not found Oct 31 03:28:22.216945 ip-10-0-134-106 kubenswrapper[2576]: E1031 03:28:22.216890 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:24.216883998 +0000 UTC m=+41.524001010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : configmap "kube-rbac-proxy" not found Oct 31 03:28:22.369827 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.369752 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq" Oct 31 03:28:22.454662 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.454625 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hnt22-469x9" event={"ID":"2d2a2662-e327-4427-a3e8-482de3a8ed26","Type":"ContainerStarted","Data":"093dbcd91688b4f46084730b8bf5451b0e6af412eaf61f864416411886017e0e"} Oct 31 03:28:22.708741 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.708708 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sjcw2"] Oct 31 03:28:22.723265 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.723237 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:22.766845 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.766809 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sjcw2"] Oct 31 03:28:22.776961 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.776935 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 03:28:22.776961 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.776937 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8tvwd\"" Oct 31 03:28:22.777145 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.776965 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 03:28:22.777145 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.777011 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 03:28:22.777145 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.777048 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 03:28:22.822716 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.822682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/abdcefed-e204-47f4-a289-5895681e12a6-signing-key\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:22.822869 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.822730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/abdcefed-e204-47f4-a289-5895681e12a6-signing-cabundle\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:22.822938 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.822861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjqdm\" (UniqueName: \"kubernetes.io/projected/abdcefed-e204-47f4-a289-5895681e12a6-kube-api-access-jjqdm\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:22.923740 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.923705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjqdm\" (UniqueName: \"kubernetes.io/projected/abdcefed-e204-47f4-a289-5895681e12a6-kube-api-access-jjqdm\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:22.923928 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.923909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/abdcefed-e204-47f4-a289-5895681e12a6-signing-key\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:22.923999 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.923937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/abdcefed-e204-47f4-a289-5895681e12a6-signing-cabundle\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:22.924719 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.924695 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/abdcefed-e204-47f4-a289-5895681e12a6-signing-cabundle\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:22.926815 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.926788 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/abdcefed-e204-47f4-a289-5895681e12a6-signing-key\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:22.942895 ip-10-0-134-106 kubenswrapper[2576]: W1031 03:28:22.942865 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d2a2ab_d78a_47c1_8e0c_84f7418d2b81.slice/crio-1685dc7421036ddfa2c95c056fc32cceb4f26686bd39cfdecfcb48955723d99e WatchSource:0}: Error finding container 1685dc7421036ddfa2c95c056fc32cceb4f26686bd39cfdecfcb48955723d99e: Status 404 returned error can't find the container with id 1685dc7421036ddfa2c95c056fc32cceb4f26686bd39cfdecfcb48955723d99e Oct 31 03:28:22.943596 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.943571 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq"] Oct 31 03:28:22.949457 ip-10-0-134-106 kubenswrapper[2576]: I1031 03:28:22.949433 2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:28:22.949612 ip-10-0-134-106 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 03:28:22.961671 ip-10-0-134-106 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 03:28:22.961938 ip-10-0-134-106 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 03:28:22.962028 ip-10-0-134-106 systemd[1]: kubelet.service: Consumed 2.383s CPU time. Oct 31 03:28:22.969689 ip-10-0-134-106 systemd[1]: Starting Kubernetes Kubelet... Oct 31 03:28:23.044382 ip-10-0-134-106 kubenswrapper[4426]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:28:23.044382 ip-10-0-134-106 kubenswrapper[4426]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 03:28:23.044382 ip-10-0-134-106 kubenswrapper[4426]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:28:23.044382 ip-10-0-134-106 kubenswrapper[4426]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:28:23.046525 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046448 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:28:23.046525 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046525 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046529 4426 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046533 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046536 4426 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046539 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046542 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046545 4426 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046548 4426 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046551 4426 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046553 4426 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046556 4426 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046559 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046563 4426 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046568 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046571 4426 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046574 4426 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046577 4426 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046579 4426 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:28:23.046597 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046583 4426 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046587 4426 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046591 4426 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046594 4426 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046596 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046599 4426 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046602 4426 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046605 4426 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046608 4426 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046611 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046613 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046616 4426 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046620 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046622 4426 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046625 4426 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046628 4426 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046631 4426 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046634 4426 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046637 4426 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046639 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:28:23.047049 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046650 4426 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046653 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046656 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046659 4426 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046662 4426 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046664 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046667 4426 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046669 4426 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046672 4426 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046675 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046678 4426 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046681 4426 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046683 4426 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046686 4426 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046689 4426 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046692 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046694 4426 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046697 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046699 4426 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046702 4426 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:28:23.047588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046704 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046707 4426 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046709 4426 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046712 4426 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046715 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046717 4426 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046719 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046722 4426 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046724 4426 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046727 4426 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046730 4426 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046732 4426 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046735 4426 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046743 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046746 4426 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046749 4426 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046751 4426 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046754 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046756 4426 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046759 4426 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:28:23.048073 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046761 4426 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046764 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046767 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046770 4426 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046772 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046775 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046778 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046781 4426 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046784 4426 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046786 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046789 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046791 4426 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046794 4426 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046796 4426 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046799 4426 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046801 4426 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046804 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046806 4426 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046809 4426 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:28:23.048585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046811 4426 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046814 4426 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046817 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046820 4426 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046823 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046825 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046828 4426 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.046836 4426 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047615 4426 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047622 4426 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047624 4426 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047627 4426 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047630 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047632 4426 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047636 4426 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047639 4426 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047641 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047644 4426 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047647 4426 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047651 4426 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:28:23.049057 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047653 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047656 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047659 4426 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047661 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047664 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047667 4426 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047669 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047672 4426 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047675 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047677 4426 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047680 4426 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047683 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047685 4426 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047688 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047690 4426 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047693 4426 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047696 4426 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047698 4426 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047701 4426 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047703 4426 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:28:23.049611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047707 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047710 4426 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047712 4426 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047715 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047718 4426 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047721 4426 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047723 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047726 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047728 4426 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047731 4426 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047734 4426 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047736 4426 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047739 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047742 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047744 4426 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047747 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047749 4426 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047752 4426 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047754 4426 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:28:23.050099 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047757 4426 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047760 4426 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047762 4426 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047765 4426 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047767 4426 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047771 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047773 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047776 4426 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047778 4426 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047781 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047783 4426 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047786 4426 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047788 4426 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047791 4426 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047794 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047797 4426 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047800 4426 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047803 4426 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047805 4426 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047808 4426 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:28:23.050595 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047810 4426 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047813 4426 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047815 4426 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047818 4426 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047820 4426 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047823 4426 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047826 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047828 4426 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047830 4426 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047833 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047835 4426 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047838 4426 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047841 4426 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047843 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047845 4426 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047848 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047851 4426 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047853 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047856 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047859 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:28:23.051138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047861 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047863 4426 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047866 4426 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047868 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047871 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047874 4426 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047878 4426 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047881 4426 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047884 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047887 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047889 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047892 4426 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047897 4426 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047901 4426 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.047904 4426 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048487 4426 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048494 4426 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048497 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048500 4426 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:28:23.051630 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048503 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048505 4426 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048508 4426 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048512 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048514 4426 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048517 4426 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048519 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048522 4426 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048525 4426 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048527 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048530 4426 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048533 4426 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048536 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048542 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048545 4426 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048547 4426 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048550 4426 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048553 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048555 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:28:23.052101 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048558 4426 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048567 4426 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048570 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048573 4426 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048575 4426 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048578 4426 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048581 4426 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048583 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048586 4426 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048588 4426 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048591 4426 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048594 4426 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048597 4426 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048600 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048602 4426 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048605 4426 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048608 4426 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048610 4426 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048612 4426 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:28:23.052601 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048615 4426 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048618 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048620 4426 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048623 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048625 4426 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048628 4426 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048630 4426 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048634 4426 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048637 4426 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048639 4426 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048642 4426 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048645 4426 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048647 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048652 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048654 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048657 4426 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048660 4426 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048662 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048665 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048667 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:28:23.053063 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048670 4426 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048672 4426 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048675 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048678 4426 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048681 4426 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048683 4426 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048686 4426 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048688 4426 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048691 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048693 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048696 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048698 4426 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048701 4426 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048703 4426 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048706 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048708 4426 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048712 4426 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048716 4426 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048718 4426 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048722 4426 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:28:23.053617 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048725 4426 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048727 4426 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048731 4426 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048735 4426 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048738 4426 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048742 4426 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048744 4426 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048747 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048756 4426 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048759 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048761 4426 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048764 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048766 4426 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048769 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048771 4426 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048774 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048776 4426 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048779 4426 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048781 4426 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:28:23.054414 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048784 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048786 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048789 4426 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048791 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.048795 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048897 4426 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048908 4426 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048918 4426 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048924 4426 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048928 4426 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048931 4426 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048936 4426 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048942 4426 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048946 4426 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048949 4426 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048952 4426 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048956 4426 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048960 4426 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048964 4426 flags.go:64] FLAG: --cgroup-root="" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048967 4426 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048970 4426 flags.go:64] FLAG: --client-ca-file="" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048973 4426 flags.go:64] FLAG: --cloud-provider="external" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048977 4426 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 03:28:23.054960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048982 4426 flags.go:64] FLAG: --cluster-domain="" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048984 4426 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048988 4426 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048991 4426 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048994 4426 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.048998 4426 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049001 4426 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049004 4426 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049007 4426 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049011 4426 flags.go:64] FLAG: --contention-profiling="false" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049014 4426 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049017 4426 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049020 4426 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049023 4426 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049027 4426 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049030 4426 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049033 4426 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049036 4426 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049039 4426 flags.go:64] FLAG: --enable-server="true" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049043 4426 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049047 4426 flags.go:64] FLAG: --event-burst="100" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049051 4426 flags.go:64] FLAG: --event-qps="50" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049054 4426 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049058 4426 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 03:28:23.055577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049061 4426 flags.go:64] FLAG: --eviction-hard="" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049066 4426 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049069 4426 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049074 4426 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049078 4426 flags.go:64] FLAG: --eviction-soft="" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049081 4426 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049084 4426 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049088 4426 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049091 4426 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049094 4426 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049097 4426 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049100 4426 flags.go:64] FLAG: --feature-gates="" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049108 4426 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049111 4426 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049114 4426 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049117 4426 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049120 4426 flags.go:64] FLAG: --healthz-port="10248" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049135 4426 flags.go:64] FLAG: --help="false" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049138 4426 flags.go:64] FLAG: --hostname-override="ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049142 4426 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049145 4426 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049148 4426 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049151 4426 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 03:28:23.056198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049155 4426 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049158 4426 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049161 4426 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049164 4426 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049167 4426 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049170 4426 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049174 4426 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049177 4426 flags.go:64] FLAG: --kube-reserved="" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049180 4426 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049183 4426 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049186 4426 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049189 4426 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049193 4426 flags.go:64] FLAG: --lock-file="" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049196 4426 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049199 4426 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049202 4426 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049207 4426 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049211 4426 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049214 4426 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049217 4426 flags.go:64] FLAG: --logging-format="text" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049220 4426 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049224 4426 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049226 4426 flags.go:64] FLAG: --manifest-url="" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049229 4426 flags.go:64] FLAG: --manifest-url-header="" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049234 4426 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 03:28:23.056743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049237 4426 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049241 4426 flags.go:64] FLAG: --max-pods="110" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049244 4426 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049247 4426 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049250 4426 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049253 4426 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049256 4426 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049259 4426 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049262 4426 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049269 4426 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049273 4426 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049276 4426 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049279 4426 flags.go:64] FLAG: --pod-cidr="" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049283 4426 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049286 4426 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049289 4426 flags.go:64] FLAG: --pods-per-core="0" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049292 4426 flags.go:64] FLAG: --port="10250" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049296 4426 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049298 4426 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0b963bd819d464b75" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049303 4426 flags.go:64] FLAG: --qos-reserved="" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049306 4426 flags.go:64] FLAG: --read-only-port="10255" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049309 4426 flags.go:64] FLAG: --register-node="true" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049312 4426 flags.go:64] FLAG: --register-with-taints="" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049316 4426 flags.go:64] FLAG: --registry-burst="10" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049319 4426 flags.go:64] FLAG: --registry-qps="5" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049322 4426 flags.go:64] FLAG: --reserved-cpus="" Oct 31 03:28:23.057468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049325 4426 flags.go:64] FLAG: --reserved-memory="" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049330 4426 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049333 4426 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049336 4426 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049339 4426 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049342 4426 flags.go:64] FLAG: --runonce="false" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049345 4426 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049348 4426 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049351 4426 flags.go:64] FLAG: --seccomp-default="false" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049354 4426 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049357 4426 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049360 4426 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049363 4426 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049366 4426 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049369 4426 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049372 4426 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049375 4426 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049378 4426 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049381 4426 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049385 4426 flags.go:64] FLAG: --system-cgroups="" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049388 4426 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049393 4426 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049396 4426 flags.go:64] FLAG: --tls-cert-file="" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049399 4426 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049403 4426 flags.go:64] FLAG: --tls-min-version="" Oct 31 03:28:23.058121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049407 4426 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049410 4426 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049413 4426 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049416 4426 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049419 4426 flags.go:64] FLAG: --v="2" Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049424 4426 flags.go:64] FLAG: --version="false" Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049428 4426 flags.go:64] FLAG: --vmodule="" Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049432 4426 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049436 4426 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049595 4426 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049599 4426 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049602 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049604 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049608 4426 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049611 4426 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049614 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049617 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049620 4426 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049623 4426 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049627 4426 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049630 4426 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:28:23.058926 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049633 4426 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049636 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049639 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049641 4426 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049644 4426 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049648 4426 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049651 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049653 4426 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049656 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049658 4426 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049661 4426 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049665 4426 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049668 4426 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049670 4426 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049673 4426 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049676 4426 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049678 4426 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049681 4426 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049685 4426 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:28:23.059478 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049689 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049693 4426 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049697 4426 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049700 4426 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049702 4426 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049705 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049707 4426 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049710 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049713 4426 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049715 4426 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049718 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049720 4426 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049723 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049725 4426 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049728 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049730 4426 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049733 4426 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049735 4426 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049739 4426 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:28:23.060105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049742 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049744 4426 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049747 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049749 4426 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049752 4426 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049756 4426 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049758 4426 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049761 4426 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049763 4426 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049766 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049768 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049771 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049774 4426 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049776 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049779 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049782 4426 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049785 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049787 4426 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049790 4426 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049792 4426 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:28:23.060703 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049795 4426 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049797 4426 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049800 4426 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049803 4426 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049805 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049808 4426 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049810 4426 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049813 4426 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049815 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049818 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049820 4426 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049824 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049827 4426 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049830 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049832 4426 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049835 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049838 4426 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049841 4426 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049844 4426 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049846 4426 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:28:23.061373 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049849 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049852 4426 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049854 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049857 4426 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049860 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049862 4426 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049865 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049867 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049870 4426 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049873 4426 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049876 4426 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049878 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049881 4426 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049883 4426 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049886 4426 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049888 4426 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:28:23.061869 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.049915 4426 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049948 4426 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049952 4426 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049956 4426 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049958 4426 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049961 4426 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049965 4426 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049969 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049973 4426 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049976 4426 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049979 4426 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049981 4426 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049984 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049988 4426 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049991 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049993 4426 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049996 4426 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.049999 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050001 4426 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:28:23.062267 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050004 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050006 4426 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050009 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050012 4426 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050014 4426 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050016 4426 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050019 4426 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050022 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050025 4426 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050027 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050030 4426 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050032 4426 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050035 4426 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050037 4426 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050040 4426 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050042 4426 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050045 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050047 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050050 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050053 4426 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:28:23.062811 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050055 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050059 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050061 4426 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050064 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050067 4426 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050069 4426 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050073 4426 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050076 4426 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050079 4426 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050081 4426 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050084 4426 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050086 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050089 4426 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050092 4426 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050094 4426 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050097 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050099 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050102 4426 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050104 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050109 4426 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:28:23.063472 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050112 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050115 4426 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050117 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050120 4426 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050153 4426 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050156 4426 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050158 4426 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050161 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050163 4426 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050166 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050168 4426 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050171 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050174 4426 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050177 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050180 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050182 4426 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050185 4426 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050187 4426 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050190 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:28:23.064010 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050193 4426 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050196 4426 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050198 4426 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050201 4426 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050203 4426 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050206 4426 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050208 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050211 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050213 4426 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050216 4426 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050218 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050221 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050223 4426 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050226 4426 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050231 4426 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050234 4426 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050237 4426 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050240 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050242 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050245 4426 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:28:23.064505 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050248 4426 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050250 4426 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050252 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050255 4426 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050258 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050260 4426 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050263 4426 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050265 4426 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.050268 4426 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.055321 4426 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.055337 4426 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.055363 4426 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.055369 4426 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055429 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055434 4426 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055437 4426 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055440 4426 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055443 4426 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055446 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:28:23.064979 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055449 4426 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055452 4426 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055455 4426 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055458 4426 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055460 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055463 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055466 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055469 4426 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055472 4426 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055475 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055477 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055481 4426 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055485 4426 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055488 4426 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055492 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055494 4426 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055497 4426 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055500 4426 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055502 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:28:23.065467 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055505 4426 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055508 4426 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055515 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055519 4426 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055522 4426 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055525 4426 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055528 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055531 4426 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055533 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055536 4426 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055539 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055541 4426 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055544 4426 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055547 4426 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055549 4426 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055552 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055555 4426 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055557 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055560 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:28:23.065920 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055563 4426 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055566 4426 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055569 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055572 4426 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055575 4426 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055578 4426 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055581 4426 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055584 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055586 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055589 4426 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055592 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055594 4426 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055597 4426 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055600 4426 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055603 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055605 4426 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055609 4426 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055611 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055614 4426 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055617 4426 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:28:23.066433 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055620 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055622 4426 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055625 4426 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055627 4426 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055630 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055632 4426 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055635 4426 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055638 4426 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055641 4426 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055643 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055646 4426 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055649 4426 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055651 4426 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055654 4426 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055656 4426 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055660 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055662 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055665 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055668 4426 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055670 4426 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:28:23.066908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055673 4426 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055676 4426 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055678 4426 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055681 4426 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055684 4426 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055686 4426 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055689 4426 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055692 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055694 4426 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055697 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055700 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055702 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055705 4426 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055707 4426 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055710 4426 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055712 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055715 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055718 4426 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055720 4426 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:28:23.067395 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055723 4426 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055727 4426 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055731 4426 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.055761 4426 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055900 4426 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055906 4426 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055909 4426 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055912 4426 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055915 4426 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055918 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055921 4426 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055924 4426 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055926 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055929 4426 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055932 4426 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:28:23.067952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055934 4426 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055938 4426 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055943 4426 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055946 4426 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055949 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055952 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055954 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055958 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055960 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055963 4426 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055966 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055968 4426 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055971 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055973 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055976 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055979 4426 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055981 4426 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055984 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055986 4426 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:28:23.068445 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055989 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055992 4426 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055994 4426 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.055997 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056000 4426 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056002 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056005 4426 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056008 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056011 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056014 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056016 4426 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056019 4426 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056022 4426 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056024 4426 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056027 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056029 4426 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056032 4426 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056035 4426 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056037 4426 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:28:23.068908 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056040 4426 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056042 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056045 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056048 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056050 4426 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056053 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056055 4426 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056058 4426 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056060 4426 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056063 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056065 4426 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056068 4426 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056070 4426 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056072 4426 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056075 4426 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056077 4426 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056080 4426 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056084 4426 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056087 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:28:23.069389 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056090 4426 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056093 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056096 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056103 4426 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056106 4426 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056109 4426 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056121 4426 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056137 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056140 4426 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056143 4426 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056146 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056148 4426 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056151 4426 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056154 4426 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056157 4426 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056159 4426 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056162 4426 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056165 4426 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056168 4426 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056170 4426 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:28:23.069843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056173 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056176 4426 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056178 4426 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056181 4426 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056184 4426 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056187 4426 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056189 4426 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056192 4426 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056195 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056197 4426 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056200 4426 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056203 4426 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056205 4426 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056208 4426 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056211 4426 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056218 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056221 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:28:23.070338 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056225 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.056250 4426 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056289 4426 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056294 4426 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056297 4426 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056301 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056304 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056307 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056309 4426 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056312 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056314 4426 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056317 4426 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056320 4426 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056322 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056325 4426 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056327 4426 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:28:23.070747 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056330 4426 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056332 4426 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056335 4426 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056338 4426 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056340 4426 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056344 4426 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056348 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056351 4426 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056354 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056357 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056361 4426 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056363 4426 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056366 4426 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056368 4426 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056372 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056377 4426 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056380 4426 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056383 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056385 4426 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056388 4426 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:28:23.071170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056391 4426 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056394 4426 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056397 4426 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056399 4426 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056402 4426 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056404 4426 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056407 4426 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056410 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056412 4426 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056415 4426 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056417 4426 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056420 4426 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056423 4426 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056425 4426 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056428 4426 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056431 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056433 4426 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056436 4426 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056438 4426 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056441 4426 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:28:23.071856 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056443 4426 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056447 4426 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056449 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056452 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056455 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056457 4426 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056462 4426 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056466 4426 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056469 4426 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056473 4426 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056475 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056478 4426 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056480 4426 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056483 4426 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056485 4426 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056488 4426 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056490 4426 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056495 4426 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056499 4426 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:28:23.072552 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056501 4426 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056504 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056507 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056509 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056512 4426 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056515 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056517 4426 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056520 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056522 4426 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056525 4426 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056527 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056530 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056533 4426 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056535 4426 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056538 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056540 4426 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056543 4426 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056545 4426 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056548 4426 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:28:23.073043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056552 4426 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056557 4426 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056560 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056562 4426 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056566 4426 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056568 4426 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056571 4426 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056574 4426 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056576 4426 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056579 4426 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056581 4426 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056584 4426 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056586 4426 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:23.056589 4426 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.056640 4426 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.057622 4426 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.057685 4426 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.057858 4426 server.go:1014] "Starting client certificate rotation" Oct 31 03:28:23.073611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.058006 4426 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:28:23.074076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.058047 4426 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:22:43 +0000 UTC" deadline="2028-08-12 17:51:08.762706154 +0000 UTC" Oct 31 03:28:23.074076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.058087 4426 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15638h22m45.704621262s" Oct 31 03:28:23.074076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.058333 4426 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:28:23.074076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.059464 4426 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:28:23.074076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.060891 4426 log.go:25] "Validated CRI v1 runtime API" Oct 31 03:28:23.074076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.062471 4426 log.go:25] "Validated CRI v1 image API" Oct 31 03:28:23.074076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.063217 4426 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 03:28:23.074076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.064739 4426 fs.go:135] Filesystem UUIDs: map[2a199e6e-6ab4-4b13-91c4-0f5097e5af7d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cf1e70e4-adaa-4c97-a7a0-211a11de3f22:/dev/nvme0n1p3] Oct 31 03:28:23.074374 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.064759 4426 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00747817d41ac8c6c61135aa78011e89d041d1bf49759cfff7aaffda9f9c3be2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00747817d41ac8c6c61135aa78011e89d041d1bf49759cfff7aaffda9f9c3be2/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/093dbcd91688b4f46084730b8bf5451b0e6af412eaf61f864416411886017e0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/093dbcd91688b4f46084730b8bf5451b0e6af412eaf61f864416411886017e0e/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1685dc7421036ddfa2c95c056fc32cceb4f26686bd39cfdecfcb48955723d99e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1685dc7421036ddfa2c95c056fc32cceb4f26686bd39cfdecfcb48955723d99e/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32d421ad4cf8e978e969ee60350ce7c259cd11aab322e053b6e7920de73497c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32d421ad4cf8e978e969ee60350ce7c259cd11aab322e053b6e7920de73497c6/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39a17dd3b5102d0eb14f9984185641c1a39b5199b4f11f9f42f42c8521408719/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39a17dd3b5102d0eb14f9984185641c1a39b5199b4f11f9f42f42c8521408719/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43c43b2135c93e9451676dc9f787280ef50687f9576b4c7c534716851e166aa8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43c43b2135c93e9451676dc9f787280ef50687f9576b4c7c534716851e166aa8/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c323437e958654bbc17871bc5fbdbcbe6c0ccf05d539eb833181ddb4fecff4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c323437e958654bbc17871bc5fbdbcbe6c0ccf05d539eb833181ddb4fecff4a/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a498898d9587d10dcdfacda5b01fe6b4bbb138bd8ba2c78e66c9e6b6bfcc8b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a498898d9587d10dcdfacda5b01fe6b4bbb138bd8ba2c78e66c9e6b6bfcc8b0/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7cca36f210d4a8551823b7e0b90e3d8160ce272085ac73c1722b039719568e6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7cca36f210d4a8551823b7e0b90e3d8160ce272085ac73c1722b039719568e6b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d02727ef543b5eace85884e096e8f1f3500c67b05de6ef37074a7c06c9235a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d02727ef543b5eace85884e096e8f1f3500c67b05de6ef37074a7c06c9235a5/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95281851c410cc7594f00a32f18b184d78b032a186a5d16a25e5c0eb854a1004/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95281851c410cc7594f00a32f18b184d78b032a186a5d16a25e5c0eb854a1004/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3244e8d47504d7f3c3dc14d767806c746ec460ad7e15abef23e3e5be7d528d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3244e8d47504d7f3c3dc14d767806c746ec460ad7e15abef23e3e5be7d528d8/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf51eab920a360395e0836861207f7b44d760a326ed20f7860e998a0cfd1befe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf51eab920a360395e0836861207f7b44d760a326ed20f7860e998a0cfd1befe/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbb070557619b8b33857fdef7b5d6cd213390898069caf0044daf30a84a0ae94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbb070557619b8b33857fdef7b5d6cd213390898069caf0044daf30a84a0ae94/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e09b04ce16de788749087c777bf051dc1de26c2b669f1ba82f56e189cc523a82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e09b04ce16de788749087c777bf051dc1de26c2b669f1ba82f56e189cc523a82/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14d2a2ab-d78a-47c1-8e0c-84f7418d2b81/volumes/kubernetes.io~projected/kube-api-access-c5hs6:{mountpoint:/var/lib/kubelet/pods/14d2a2ab-d78a-47c1-8e0c-84f7418d2b81/volumes/kubernetes.io~projected/kube-api-access-c5hs6 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/288945de-f0e7-47c7-87c9-45f8dbc021d9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/288945de-f0e7-47c7-87c9-45f8dbc021d9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/288945de-f0e7-47c7-87c9-45f8dbc021d9/volumes/kubernetes.io~projected/kube-api-access-xcs67:{mountpoint:/var/lib/kubelet/pods/288945de-f0e7-47c7-87c9-45f8dbc021d9/volumes/kubernetes.io~projected/kube-api-access-xcs67 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/288945de-f0e7-47c7-87c9-45f8dbc021d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/288945de-f0e7-47c7-87c9-45f8dbc021d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28de4e55-40b4-4721-9f39-2bbd1033c06e/volumes/kubernetes.io~projected/kube-api-access-zdfsd:{mountpoint:/var/lib/kubelet/pods/28de4e55-40b4-4721-9f39-2bbd1033c06e/volumes/kubernetes.io~projected/kube-api-access-zdfsd major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d2a2662-e327-4427-a3e8-482de3a8ed26/volumes/kubernetes.io~projected/kube-api-access-4dkdj:{mountpoint:/var/lib/kubelet/pods/2d2a2662-e327-4427-a3e8-482de3a8ed26/volumes/kubernetes.io~projected/kube-api-access-4dkdj major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0/volumes/kubernetes.io~projected/kube-api-access-hrtjc:{mountpoint:/var/lib/kubelet/pods/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0/volumes/kubernetes.io~projected/kube-api-access-hrtjc major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3dd3e64a-9a74-4542-b09a-7003e749b44f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3dd3e64a-9a74-4542-b09a-7003e749b44f/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e9ea6af-1755-4e15-861b-5740354972f2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6e9ea6af-1755-4e15-861b-5740354972f2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abdcefed-e204-47f4-a289-5895681e12a6/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/abdcefed-e204-47f4-a289-5895681e12a6/volumes/kubernetes.io~secret/signing-key major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac082af6-5ee0-4995-9636-8271debb950a/volumes/kubernetes.io~projected/kube-api-access-kgl9x:{mountpoint:/var/lib/kubelet/pods/ac082af6-5ee0-4995-9636-8271debb950a/volumes/kubernetes.io~projected/kube-api-access-kgl9x major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad4db9a5-414b-4e4c-a712-e30ae75254c6/volumes/kubernetes.io~projected/kube-api-access-w4jzx:{mountpoint:/var/lib/kubelet/pods/ad4db9a5-414b-4e4c-a712-e30ae75254c6/volumes/kubernetes.io~projected/kube-api-access-w4jzx major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49/volumes/kubernetes.io~projected/kube-api-access-dq8vh:{mountpoint:/var/lib/kubelet/pods/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49/volumes/kubernetes.io~projected/kube-api-access-dq8vh major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce341b3b-e888-41f3-90b2-2055a4634a2d/volumes/kubernetes.io~projected/kube-api-access-rb6qd:{mountpoint:/var/lib/kubelet/pods/ce341b3b-e888-41f3-90b2-2055a4634a2d/volumes/kubernetes.io~projected/kube-api-access-rb6qd major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce341b3b-e888-41f3-90b2-2055a4634a2d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ce341b3b-e888-41f3-90b2-2055a4634a2d/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e/volumes/kubernetes.io~projected/kube-api-access-jkpjn:{mountpoint:/var/lib/kubelet/pods/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e/volumes/kubernetes.io~projected/kube-api-access-jkpjn major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da393ceb-f1dc-4cb1-b82f-933a8eaa8497/volumes/kubernetes.io~projected/kube-api-access-gz6fg:{mountpoint:/var/lib/kubelet/pods/da393ceb-f1dc-4cb1-b82f-933a8eaa8497/volumes/kubernetes.io~projected/kube-api-access-gz6fg major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7bf98c9-3458-4936-8dfd-32faa8e522b9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f7bf98c9-3458-4936-8dfd-32faa8e522b9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7bf98c9-3458-4936-8dfd-32faa8e522b9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f7bf98c9-3458-4936-8dfd-32faa8e522b9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7bf98c9-3458-4936-8dfd-32faa8e522b9/volumes/kubernetes.io~projected/kube-api-access-mdqlc:{mountpoint:/var/lib/kubelet/pods/f7bf98c9-3458-4936-8dfd-32faa8e522b9/volumes/kubernetes.io~projected/kube-api-access-mdqlc major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb187718-37b3-47fa-8c44-cd650c0185be/volumes/kubernetes.io~projected/kube-api-access-nmkwg:{mountpoint:/var/lib/kubelet/pods/fb187718-37b3-47fa-8c44-cd650c0185be/volumes/kubernetes.io~projected/kube-api-access-nmkwg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc1bac22-a12b-4777-989e-a4c059633fd1/volumes/kubernetes.io~projected/kube-api-access-p76gl:{mountpoint:/var/lib/kubelet/pods/fc1bac22-a12b-4777-989e-a4c059633fd1/volumes/kubernetes.io~projected/kube-api-access-p76gl major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf/volumes/kubernetes.io~projected/kube-api-access-ckh87:{mountpoint:/var/lib/kubelet/pods/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf/volumes/kubernetes.io~projected/kube-api-access-ckh87 major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7d2ec84df4983f9a9d8a2fbcda4abdcbb880b70918c341cfc9a9ee6eacda9d42/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/00a636ce3bf6c45dae4a4426a21c40307b122422c5a09bbaaa2c884683ece0b1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/bbc8a18c486fb36a01198279fdc88323456386d9c803255a9c2179a357a2a844/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/bdcf986dcd252b1b701d8500c7d78d12e0aedcb9a495d3b9ccbb7c64df74423a/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/3d3e2ecb90e65f8f9db598e1bbc9b2be25a5d5ddac90b334420510c2ce4d273d/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6a9d34187b42c0a684636da0620ecf1ad8b58e995832cfab1c4db13a0cfdd33c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/56a05c81e9611558dfcb12db59b740bcdff3efdd0906dd2da39ab562aa816d3f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/30013f26f87e6de453514b025700b1ceb8462925c7ce955e58bae7a5588092bd/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c4badc0295bbb774e5b7474db22d5f1a001a097854e3a7b9a9b06dc00ba3af43/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/158450ee65ca56427ec4d0f96ea097a81989e2289d264f8037edfdf742831808/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/024d16d1b42797af91f68f4a2ab131c402894a3800f77dfd1e572618d1341224/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/4eb76f91c508973998ae9f1e495001413070334023a1396c737e1a46768c4a1f/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0502382cd45d6a53d7dab0a11fc9e9fb72efc70bae27f2f4c0b3260f9321d064/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2623875d1fe2f1ebb1ccff96c95ea36b7deb5b7577aa1d3f8be4b6777ac9eb69/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/dce07fdabcc6c733f3272dd1e985988ef68d9ae9ec4020c5ea8919756b1bdbf2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c610964cbb9de13e9df2e75c27eadb86eeed42c4d1b69fc2b57a03601860fa0a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/bb758d6b559c818f5440b7874bc0829ea43c29774ee4d1974d94bbc33fda99ae/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c14e6c918202c2b31ce43d56882e18ffae26b57642ed101853423ab57d62d407/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4c067b7f9cd746ccb7a1f6cdb63e3f573407d4208e131bfc17998c43931b15f5/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5aca726423755bc6855a20a30176a5237eb8199d870a671353c4c0dba4317998/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/94c68e4102a5a78214043fa2788f9c4defb9395fb165ea34f3401088214fa791/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/71dc84c7901be27a9f949b46327c3304ac93c25351a7d7ab66226e662a21d135/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/52ab27b2850f02ece924e5f207f6446c3750ec0446b6b5a8bee229521507dd29/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/da75a6e0fd45ccc034dac3349c38912136323a6ae95fa97c18376eb80ba65645/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/c14cbc1def5eec164cd4e240ca59daf03719ee5f14f4aade980e9f63254bbdc8/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/3df68b02fb14897c8597f8fd329e1559372ca02eb03e97cb2371a71f4d48692f/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/c6ac5e8b00c95ef9075f6882465fa9507e1fde6c403e157617cdebfcaeb19afa/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/c54193415b4e37f14618034415b7f2507553564119984c0a5ab4b520b148fafa/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/03c2bf929ef078b75d8b8522553242085368b2707940c14bd7fee51e6ca3bb72/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/f5eb3bcec1a8b8acff9257440507020c729b384e2540783f3da9e6573fb2bf30/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/b3b53ef3be3e5c00ca0925f97c84c9269bd5a6842dad0924f842178a7a692d59/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5af81293ad668310afbbc12fa9ced30191b5d03f71bc6ab4107146c183f30c7e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/19589ec89b3aa79e890353f87284202c7df4771d771ea7067911c4dc3467b268/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/9f830127f871b900775b728922953956caa1af913602af6f78a73be2e6397a11/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/fa2026fadbb9464c54f3b91ed0f8bda584357d4b3c9e5041cb2e21948143961d/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/5337c534b5fdcb40d2af74ca6dcab5b5f3e92dd9eb2e8fcc5ba1b18a964f1f8e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/9df9ab695f69054101f5d8ebf665816bc09ff7ac4b9c56975edd4969a5b30113/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ea26a4222723e0493f7efaf796d5f65d8bd76d8d116763ecd85ee88733358799/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/b129960c3c62a352c1200e28d666a64e23b9fb8aac8e6ca2e08dc46987755b0a/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.072117 4426 manager.go:217] Machine: {Timestamp:2026-10-31 03:28:23.071223785 +0000 UTC m=+0.071849004 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101268 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec217aa3ba86f35ed6ebe6fda6cd842c SystemUUID:ec217aa3-ba86-f35e-d6eb-e6fda6cd842c BootID:7839c4b0-92a9-4f4e-84f3-928144bc5271 Filesystems:[{Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/093dbcd91688b4f46084730b8bf5451b0e6af412eaf61f864416411886017e0e/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/28de4e55-40b4-4721-9f39-2bbd1033c06e/volumes/kubernetes.io~projected/kube-api-access-zdfsd DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43c43b2135c93e9451676dc9f787280ef50687f9576b4c7c534716851e166aa8/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00747817d41ac8c6c61135aa78011e89d041d1bf49759cfff7aaffda9f9c3be2/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/288945de-f0e7-47c7-87c9-45f8dbc021d9/volumes/kubernetes.io~projected/kube-api-access-xcs67 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce341b3b-e888-41f3-90b2-2055a4634a2d/volumes/kubernetes.io~projected/kube-api-access-rb6qd DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e9ea6af-1755-4e15-861b-5740354972f2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7bf98c9-3458-4936-8dfd-32faa8e522b9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ac082af6-5ee0-4995-9636-8271debb950a/volumes/kubernetes.io~projected/kube-api-access-kgl9x DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39a17dd3b5102d0eb14f9984185641c1a39b5199b4f11f9f42f42c8521408719/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/abdcefed-e204-47f4-a289-5895681e12a6/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:239 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3244e8d47504d7f3c3dc14d767806c746ec460ad7e15abef23e3e5be7d528d8/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da393ceb-f1dc-4cb1-b82f-933a8eaa8497/volumes/kubernetes.io~projected/kube-api-access-gz6fg DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7bf98c9-3458-4936-8dfd-32faa8e522b9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a498898d9587d10dcdfacda5b01fe6b4bbb138bd8ba2c78e66c9e6b6bfcc8b0/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32d421ad4cf8e978e969ee60350ce7c259cd11aab322e053b6e7920de73497c6/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d2a2662-e327-4427-a3e8-482de3a8ed26/volumes/kubernetes.io~projected/kube-api-access-4dkdj DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/288945de-f0e7-47c7-87c9-45f8dbc021d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49/volumes/kubernetes.io~projected/kube-api-access-dq8vh DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3dd3e64a-9a74-4542-b09a-7003e749b44f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e/volumes/kubernetes.io~projected/kube-api-access-jkpjn DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95281851c410cc7594f00a32f18b184d78b032a186a5d16a25e5c0eb854a1004/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c323437e958654bbc17871bc5fbdbcbe6c0ccf05d539eb833181ddb4fecff4a/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7cca36f210d4a8551823b7e0b90e3d8160ce272085ac73c1722b039719568e6b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf/volumes/kubernetes.io~projected/kube-api-access-ckh87 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d02727ef543b5eace85884e096e8f1f3500c67b05de6ef37074a7c06c9235a5/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e09b04ce16de788749087c777bf051dc1de26c2b669f1ba82f56e189cc523a82/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7bf98c9-3458-4936-8dfd-32faa8e522b9/volumes/kubernetes.io~projected/kube-api-access-mdqlc DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0/volumes/kubernetes.io~projected/kube-api-access-hrtjc DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/288945de-f0e7-47c7-87c9-45f8dbc021d9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ad4db9a5-414b-4e4c-a712-e30ae75254c6/volumes/kubernetes.io~projected/kube-api-access-w4jzx DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14d2a2ab-d78a-47c1-8e0c-84f7418d2b81/volumes/kubernetes.io~projected/kube-api-access-c5hs6 DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbb070557619b8b33857fdef7b5d6cd213390898069caf0044daf30a84a0ae94/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc1bac22-a12b-4777-989e-a4c059633fd1/volumes/kubernetes.io~projected/kube-api-access-p76gl DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf51eab920a360395e0836861207f7b44d760a326ed20f7860e998a0cfd1befe/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce341b3b-e888-41f3-90b2-2055a4634a2d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1685dc7421036ddfa2c95c056fc32cceb4f26686bd39cfdecfcb48955723d99e/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/fb187718-37b3-47fa-8c44-cd650c0185be/volumes/kubernetes.io~projected/kube-api-access-nmkwg DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:093dbcd91688b4f MacAddress:be:a8:c3:b6:d8:a7 Speed:10000 Mtu:8901} {Name:1685dc7421036dd MacAddress:42:09:07:f4:de:e9 Speed:10000 Mtu:8901} {Name:39a17dd3b5102d0 MacAddress:42:36:b6:d2:a9:d6 Speed:10000 Mtu:8901} {Name:b3244e8d47504d7 MacAddress:fe:22:ed:a9:d0:2e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e6:02:f2:35 Speed:0 Mtu:9001} {Name:br-int MacAddress:4e:f7:80:44:3a:55 Speed:0 Mtu:8901} {Name:e09b04ce16de788 MacAddress:fe:2e:8b:23:7b:76 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e6:02:f2:35 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c6:76:1a:86:9e:50 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:02:a3:5a:82:2f:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.072839 4426 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.072902 4426 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073248 4426 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073275 4426 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-106.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073394 4426 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073401 4426 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073411 4426 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073429 4426 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073493 4426 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073693 4426 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073797 4426 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073830 4426 kubelet.go:493] "Attempting to sync node with API server" Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073843 4426 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073852 4426 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073862 4426 kubelet.go:405] "Adding apiserver pod source" Oct 31 03:28:23.074518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.073870 4426 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 03:28:23.075047 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.074778 4426 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 03:28:23.075047 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.074791 4426 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:28:23.075284 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.075265 4426 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 03:28:23.081873 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.081860 4426 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 03:28:23.081873 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.081873 4426 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:28:23.081972 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.081966 4426 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 03:28:23.082173 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082161 4426 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 03:28:23.082208 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082191 4426 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 03:28:23.082291 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082282 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 03:28:23.082322 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082296 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 03:28:23.082322 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082303 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 03:28:23.082322 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082309 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 03:28:23.082322 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082315 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 03:28:23.082322 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082321 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 03:28:23.082522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082327 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 03:28:23.082522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082333 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 03:28:23.082522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082340 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 03:28:23.082522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082346 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 03:28:23.082522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082354 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 03:28:23.082522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082363 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 03:28:23.082522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082387 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 03:28:23.082522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.082393 4426 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 03:28:23.085629 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.085616 4426 server.go:1263] "Started kubelet" Oct 31 03:28:23.086116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.085708 4426 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 03:28:23.086189 ip-10-0-134-106 systemd[1]: Started Kubernetes Kubelet. Oct 31 03:28:23.086360 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.086272 4426 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 03:28:23.087446 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.087418 4426 server.go:317] "Adding debug handlers to kubelet server" Oct 31 03:28:23.087884 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.085845 4426 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 03:28:23.088296 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.088266 4426 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 03:28:23.089307 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.089289 4426 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 03:28:23.089416 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.089333 4426 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 03:28:23.089416 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.089408 4426 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 03:28:23.089416 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.089417 4426 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 03:28:23.089689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.089409 4426 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:22:44 +0000 UTC" deadline="2028-05-19 07:25:31.20998223 +0000 UTC" Oct 31 03:28:23.089758 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.089710 4426 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13587h57m8.120279067s" Oct 31 03:28:23.090839 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:23.090804 4426 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-106.us-west-2.compute.internal\" not found" Oct 31 03:28:23.091058 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.091041 4426 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 03:28:23.091587 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.091569 4426 factory.go:55] Registering systemd factory Oct 31 03:28:23.091671 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.091589 4426 factory.go:223] Registration of the systemd container factory successfully Oct 31 03:28:23.092046 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.092027 4426 factory.go:153] Registering CRI-O factory Oct 31 03:28:23.092140 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.092050 4426 factory.go:223] Registration of the crio container factory successfully Oct 31 03:28:23.092140 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.092121 4426 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 03:28:23.092236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.092152 4426 factory.go:103] Registering Raw factory Oct 31 03:28:23.092236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.092167 4426 manager.go:1199] Started watching for new ooms in manager Oct 31 03:28:23.092500 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.092487 4426 manager.go:319] Starting recovery of all containers Oct 31 03:28:23.093013 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.092973 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49" volumeName="kubernetes.io/projected/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-kube-api-access-dq8vh" seLinuxMountContext="" Oct 31 03:28:23.093073 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093018 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7bf98c9-3458-4936-8dfd-32faa8e522b9" volumeName="kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-tmp" seLinuxMountContext="" Oct 31 03:28:23.093073 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093041 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb187718-37b3-47fa-8c44-cd650c0185be" volumeName="kubernetes.io/projected/fb187718-37b3-47fa-8c44-cd650c0185be-kube-api-access-nmkwg" seLinuxMountContext="" Oct 31 03:28:23.093073 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093057 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14d2a2ab-d78a-47c1-8e0c-84f7418d2b81" volumeName="kubernetes.io/projected/14d2a2ab-d78a-47c1-8e0c-84f7418d2b81-kube-api-access-c5hs6" seLinuxMountContext="" Oct 31 03:28:23.093246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093076 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28de4e55-40b4-4721-9f39-2bbd1033c06e" volumeName="kubernetes.io/projected/28de4e55-40b4-4721-9f39-2bbd1033c06e-kube-api-access-zdfsd" seLinuxMountContext="" Oct 31 03:28:23.093246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093092 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0" volumeName="kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-daemon-config" seLinuxMountContext="" Oct 31 03:28:23.093246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093107 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3dd3e64a-9a74-4542-b09a-7003e749b44f" volumeName="kubernetes.io/secret/3dd3e64a-9a74-4542-b09a-7003e749b44f-agent-certs" seLinuxMountContext="" Oct 31 03:28:23.093246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093140 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad4db9a5-414b-4e4c-a712-e30ae75254c6" volumeName="kubernetes.io/projected/ad4db9a5-414b-4e4c-a712-e30ae75254c6-kube-api-access-w4jzx" seLinuxMountContext="" Oct 31 03:28:23.093246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093161 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="288945de-f0e7-47c7-87c9-45f8dbc021d9" volumeName="kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-env-overrides" seLinuxMountContext="" Oct 31 03:28:23.093246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093181 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da393ceb-f1dc-4cb1-b82f-933a8eaa8497" volumeName="kubernetes.io/empty-dir/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-data-volume" seLinuxMountContext="" Oct 31 03:28:23.093246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093196 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7bf98c9-3458-4936-8dfd-32faa8e522b9" volumeName="kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-tuned" seLinuxMountContext="" Oct 31 03:28:23.093246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093217 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2a2662-e327-4427-a3e8-482de3a8ed26" volumeName="kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-service-ca-bundle" seLinuxMountContext="" Oct 31 03:28:23.093246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093232 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0" volumeName="kubernetes.io/projected/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-kube-api-access-hrtjc" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093247 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac082af6-5ee0-4995-9636-8271debb950a" volumeName="kubernetes.io/configmap/ac082af6-5ee0-4995-9636-8271debb950a-serviceca" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093286 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce341b3b-e888-41f3-90b2-2055a4634a2d" volumeName="kubernetes.io/secret/ce341b3b-e888-41f3-90b2-2055a4634a2d-cert" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093305 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc1bac22-a12b-4777-989e-a4c059633fd1" volumeName="kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093318 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc3ecb5-d1e9-4125-a700-7c4b992b75bf" volumeName="kubernetes.io/projected/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-kube-api-access-ckh87" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093334 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28de4e55-40b4-4721-9f39-2bbd1033c06e" volumeName="kubernetes.io/configmap/28de4e55-40b4-4721-9f39-2bbd1033c06e-iptables-alerter-script" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093351 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abdcefed-e204-47f4-a289-5895681e12a6" volumeName="kubernetes.io/configmap/abdcefed-e204-47f4-a289-5895681e12a6-signing-cabundle" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093363 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad4db9a5-414b-4e4c-a712-e30ae75254c6" volumeName="kubernetes.io/configmap/ad4db9a5-414b-4e4c-a712-e30ae75254c6-config-volume" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093380 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad4db9a5-414b-4e4c-a712-e30ae75254c6" volumeName="kubernetes.io/empty-dir/ad4db9a5-414b-4e4c-a712-e30ae75254c6-tmp-dir" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093393 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49" volumeName="kubernetes.io/empty-dir/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-tmp-dir" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093405 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7bf98c9-3458-4936-8dfd-32faa8e522b9" volumeName="kubernetes.io/projected/f7bf98c9-3458-4936-8dfd-32faa8e522b9-kube-api-access-mdqlc" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093426 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb187718-37b3-47fa-8c44-cd650c0185be" volumeName="kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-binary-copy" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093439 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb187718-37b3-47fa-8c44-cd650c0185be" volumeName="kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093457 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="288945de-f0e7-47c7-87c9-45f8dbc021d9" volumeName="kubernetes.io/secret/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093470 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="288945de-f0e7-47c7-87c9-45f8dbc021d9" volumeName="kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-config" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093484 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="288945de-f0e7-47c7-87c9-45f8dbc021d9" volumeName="kubernetes.io/projected/288945de-f0e7-47c7-87c9-45f8dbc021d9-kube-api-access-xcs67" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093510 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3dd3e64a-9a74-4542-b09a-7003e749b44f" volumeName="kubernetes.io/configmap/3dd3e64a-9a74-4542-b09a-7003e749b44f-konnectivity-ca" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093531 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e9ea6af-1755-4e15-861b-5740354972f2" volumeName="kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093550 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abdcefed-e204-47f4-a289-5895681e12a6" volumeName="kubernetes.io/secret/abdcefed-e204-47f4-a289-5895681e12a6-signing-key" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093565 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac082af6-5ee0-4995-9636-8271debb950a" volumeName="kubernetes.io/projected/ac082af6-5ee0-4995-9636-8271debb950a-kube-api-access-kgl9x" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093583 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce341b3b-e888-41f3-90b2-2055a4634a2d" volumeName="kubernetes.io/projected/ce341b3b-e888-41f3-90b2-2055a4634a2d-kube-api-access-rb6qd" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093599 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7c37d3a-09c1-48a2-bd7b-abf3975ca24e" volumeName="kubernetes.io/projected/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kube-api-access-jkpjn" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093618 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2a2662-e327-4427-a3e8-482de3a8ed26" volumeName="kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-trusted-ca-bundle" seLinuxMountContext="" Oct 31 03:28:23.093633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093633 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da393ceb-f1dc-4cb1-b82f-933a8eaa8497" volumeName="kubernetes.io/projected/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-api-access-gz6fg" seLinuxMountContext="" Oct 31 03:28:23.094675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093647 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb187718-37b3-47fa-8c44-cd650c0185be" volumeName="kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 03:28:23.094675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093666 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="288945de-f0e7-47c7-87c9-45f8dbc021d9" volumeName="kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-script-lib" seLinuxMountContext="" Oct 31 03:28:23.094675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093681 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2a2662-e327-4427-a3e8-482de3a8ed26" volumeName="kubernetes.io/empty-dir/2d2a2662-e327-4427-a3e8-482de3a8ed26-archives-path" seLinuxMountContext="" Oct 31 03:28:23.094675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093697 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2a2662-e327-4427-a3e8-482de3a8ed26" volumeName="kubernetes.io/projected/2d2a2662-e327-4427-a3e8-482de3a8ed26-kube-api-access-4dkdj" seLinuxMountContext="" Oct 31 03:28:23.094675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093709 4426 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0" volumeName="kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cni-binary-copy" seLinuxMountContext="" Oct 31 03:28:23.094675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093720 4426 reconstruct.go:97] "Volume reconstruction finished" Oct 31 03:28:23.094675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.093729 4426 reconciler.go:29] "Reconciler: start to sync state" Oct 31 03:28:23.119675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.119640 4426 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 03:28:23.121093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.121064 4426 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 03:28:23.121093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.121096 4426 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 03:28:23.121277 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.121117 4426 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 03:28:23.121277 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:23.121170 4426 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 03:28:23.140923 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.140891 4426 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="80cafb369a21f6579d3e631174bdb2f0cc91b307d1e87076f2dc8847212a25ad" exitCode=0 Oct 31 03:28:23.141032 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.140921 4426 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="221b09445b5de3846ca20cc8ea5a6c5af83aded84e60276880c7079cff5fb61a" exitCode=0 Oct 31 03:28:23.141032 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.140946 4426 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="e67324b77856497ea641ddda29022dd487b25f71c54d0f6b5ebd361df9b113fc" exitCode=0 Oct 31 03:28:23.141032 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.140954 4426 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="07e5d03f4e2835eb6f14f7dd49454fae88f00c8723e9104604484512b1d277c3" exitCode=0 Oct 31 03:28:23.141032 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.140960 4426 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="ac35a60f58205cce2789cef0093b27b2bd6fce80cdf95d137ed8ce1fe4062b19" exitCode=0 Oct 31 03:28:23.141032 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.140966 4426 generic.go:356] "Generic (PLEG): container finished" podID="fb187718-37b3-47fa-8c44-cd650c0185be" containerID="9ba8cbf7ee466215fb50fa1b0261e3e8fb5eb5e1f98acb467caecc348f9e66cc" exitCode=0 Oct 31 03:28:23.147263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.147230 4426 generic.go:356] "Generic (PLEG): container finished" podID="2625f366498146fdc680add27e4d568b" containerID="d9a9779dc13e6c7deae08f64daf66f6acda9514da28258e41a06865b4758ec05" exitCode=0 Oct 31 03:28:23.156301 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.156281 4426 manager.go:324] Recovery completed Oct 31 03:28:23.171041 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.171015 4426 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:28:23.172441 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.172417 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:28:23.172441 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.172446 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:28:23.172598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.172457 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:28:23.173945 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.173900 4426 cpu_manager.go:226] "Starting" policy="none" Oct 31 03:28:23.173945 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.173911 4426 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:28:23.174109 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.173912 4426 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 03:28:23.174109 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.173991 4426 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 03:28:23.174233 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.174214 4426 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 03:28:23.174293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.174234 4426 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 03:28:23.174293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.174259 4426 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 03:28:23.174293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.174269 4426 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 03:28:23.174293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.174277 4426 policy_none.go:50] "Start" Oct 31 03:28:23.174293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.174291 4426 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 03:28:23.174520 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.174305 4426 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 03:28:23.174967 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.174954 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:28:23.175009 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.174981 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:28:23.175009 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.174991 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:28:23.186481 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.186457 4426 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 03:28:23.186481 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.186483 4426 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 03:28:23.186702 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.186493 4426 policy_none.go:44] "Start" Oct 31 03:28:23.190647 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.190615 4426 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 03:28:23.191185 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:23.191165 4426 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-106.us-west-2.compute.internal\" not found" Oct 31 03:28:23.195234 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.195213 4426 manager.go:341] "Starting Device Plugin manager" Oct 31 03:28:23.195327 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:23.195254 4426 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 03:28:23.195327 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.195268 4426 server.go:87] "Starting device plugin registration server" Oct 31 03:28:23.195632 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.195617 4426 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 03:28:23.195676 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.195636 4426 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 03:28:23.195800 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.195783 4426 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 03:28:23.195913 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.195882 4426 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:28:23.195974 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.195919 4426 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 03:28:23.195974 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.195924 4426 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 03:28:23.196066 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.195968 4426 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:28:23.195910481Z","UUID":"23983cb3-0bde-4123-a71e-caa4a2bf3227","Handler":null,"Name":"","Endpoint":""} Oct 31 03:28:23.197278 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:23.197249 4426 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 03:28:23.197397 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:23.197294 4426 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-106.us-west-2.compute.internal\" not found" Oct 31 03:28:23.197470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.197450 4426 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 03:28:23.197558 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.197476 4426 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:28:23.222335 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.222273 4426 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal"] Oct 31 03:28:23.222499 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.222426 4426 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:28:23.223578 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.223559 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:28:23.223682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.223596 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:28:23.223682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.223608 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:28:23.223844 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.223834 4426 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:28:23.224007 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.223992 4426 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:28:23.224750 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.224733 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:28:23.224826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.224761 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:28:23.224826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.224772 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:28:23.224826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.224810 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:28:23.224917 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.224841 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:28:23.224917 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.224864 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:28:23.225049 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.225019 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" event={"ID":"2625f366498146fdc680add27e4d568b","Type":"ContainerStarted","Data":"9d10f036909470030826cbaaf856fb4d395479533d3159a0cd77881c9a6e4ea5"} Oct 31 03:28:23.225084 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.225062 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" event={"ID":"2625f366498146fdc680add27e4d568b","Type":"ContainerDied","Data":"d9a9779dc13e6c7deae08f64daf66f6acda9514da28258e41a06865b4758ec05"} Oct 31 03:28:23.225115 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.225082 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" event={"ID":"2625f366498146fdc680add27e4d568b","Type":"ContainerStarted","Data":"95281851c410cc7594f00a32f18b184d78b032a186a5d16a25e5c0eb854a1004"} Oct 31 03:28:23.225165 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:23.225037 4426 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-106.us-west-2.compute.internal\" not found" node="ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.225165 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.225111 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" event={"ID":"506bf97362686c30bca7e209bcab60a8","Type":"ContainerStarted","Data":"bc4df7d9d76057e118f61bd8a22bea793611057b24a01f8dc4842016f2de7d84"} Oct 31 03:28:23.225165 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.225157 4426 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:28:23.225249 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.225187 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" event={"ID":"506bf97362686c30bca7e209bcab60a8","Type":"ContainerStarted","Data":"43c43b2135c93e9451676dc9f787280ef50687f9576b4c7c534716851e166aa8"} Oct 31 03:28:23.226269 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.226251 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:28:23.226351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.226278 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:28:23.226351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.226288 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:28:23.226432 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:23.226397 4426 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-106.us-west-2.compute.internal\" not found" node="ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.228957 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.228940 4426 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:28:23.295906 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.295873 4426 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:28:23.297023 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.297002 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:28:23.297182 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.297041 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:28:23.297182 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.297057 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:28:23.297182 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.297090 4426 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.318808 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.318776 4426 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:28:23.321670 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.321647 4426 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:28:23.322104 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.322084 4426 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:28:23.322234 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.322116 4426 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 03:28:23.391797 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.391756 4426 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.399739 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.399711 4426 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.399845 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.399792 4426 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.400394 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.400375 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeNotReady" Oct 31 03:28:23.400504 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.400414 4426 setters.go:546] "Node became not ready" node="ip-10-0-134-106.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T03:28:23Z","lastTransitionTime":"2026-10-31T03:28:23Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 31 03:28:23.422679 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.422644 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.422679 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.422680 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.422860 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.422698 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/506bf97362686c30bca7e209bcab60a8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"506bf97362686c30bca7e209bcab60a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.428946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.428881 4426 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-106.us-west-2.compute.internal" is owned by different node (oldNodeID="7364d427-3afa-426a-ae5f-a34608b9c9c5", newNodeID=""), cleaning up... Oct 31 03:28:23.453882 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.453852 4426 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:28:23.453987 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:23.453903 4426 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.453987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.453919 4426 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.523184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.523117 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.523333 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.523192 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.523333 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.523213 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/506bf97362686c30bca7e209bcab60a8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"506bf97362686c30bca7e209bcab60a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.523333 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.523156 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.523333 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.523250 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/506bf97362686c30bca7e209bcab60a8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"506bf97362686c30bca7e209bcab60a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.523333 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.523309 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2625f366498146fdc680add27e4d568b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" (UID: \"2625f366498146fdc680add27e4d568b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.566165 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.566142 4426 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:28:23.566309 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:23.566191 4426 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:23.666813 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.666785 4426 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-106.us-west-2.compute.internal" event="NodeReady" Oct 31 03:28:23.666955 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:23.666877 4426 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 03:28:24.074801 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.074763 4426 apiserver.go:52] "Watching apiserver" Oct 31 03:28:24.159120 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.159092 4426 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:24.159272 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.159253 4426 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:24.172119 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.172087 4426 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 03:28:24.172589 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.172565 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal","openshift-multus/multus-9sx4r","kube-system/global-pull-secret-syncer-n9rx6","openshift-multus/multus-additional-cni-plugins-7txzf","openshift-service-ca/service-ca-5fffc97bdf-sjcw2","openshift-ovn-kubernetes/ovnkube-node-4hh72","openshift-cluster-node-tuning-operator/tuned-v5d8b","openshift-image-registry/node-ca-fb4pz","openshift-multus/network-metrics-daemon-rh4rl","openshift-network-operator/iptables-alerter-lt92b","openshift-dns/node-resolver-2vww2","openshift-network-diagnostics/network-check-target-bh9xh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428","openshift-ingress-canary/ingress-canary-pr54w","openshift-insights/periodic-gathering-hnt22-469x9","kube-system/konnectivity-agent-zmcjg","openshift-dns/dns-default-5rfb8","openshift-insights/insights-runtime-extractor-lhsn4","kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq"] Oct 31 03:28:24.175198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.175173 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:24.179539 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.179517 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:24.180422 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.180402 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.181232 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.181212 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:24.192484 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.192460 4426 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 03:28:24.216312 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.216289 4426 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:28:24.216398 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.216348 4426 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:24.216877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.216866 4426 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:28:24.216919 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.216895 4426 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-106.us-west-2.compute.internal" Oct 31 03:28:24.222082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.222055 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.222082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.222061 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.222248 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.222055 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.222248 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.222066 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 03:28:24.222679 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.222662 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dlcmg\"" Oct 31 03:28:24.222846 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.222828 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7b9lc\"" Oct 31 03:28:24.222960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.222854 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-8f2x4\"" Oct 31 03:28:24.222960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.222922 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 03:28:24.222960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.222946 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hztlx\"" Oct 31 03:28:24.222960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.222959 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:28:24.228442 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228422 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqlc\" (UniqueName: \"kubernetes.io/projected/f7bf98c9-3458-4936-8dfd-32faa8e522b9-kube-api-access-mdqlc\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.228502 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228451 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28de4e55-40b4-4721-9f39-2bbd1033c06e-host-slash\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:28:24.228502 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228470 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcs67\" (UniqueName: \"kubernetes.io/projected/288945de-f0e7-47c7-87c9-45f8dbc021d9-kube-api-access-xcs67\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.228502 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228487 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-data-volume\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.228619 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228503 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-bin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.228619 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228587 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-data-volume\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.228711 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228625 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-daemon-config\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.228711 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228644 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtjc\" (UniqueName: \"kubernetes.io/projected/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-kube-api-access-hrtjc\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.228711 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228664 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-var-lib-kubelet\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.228711 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228683 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckh87\" (UniqueName: \"kubernetes.io/projected/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-kube-api-access-ckh87\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:24.228711 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228700 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-tmp-dir\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:28:24.228931 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228730 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-k8s-cni-cncf-io\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.228931 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228849 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.228931 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228873 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-tmp-dir\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:28:24.229061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228967 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.229061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.228998 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysconfig\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.229061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229051 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:24.229178 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229062 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-daemon-config\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.229178 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229073 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-system-cni-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.229178 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229091 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.229178 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229118 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-socket-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.229178 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229159 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-kubelet-config\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:24.229483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229190 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-tmp\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.229483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229207 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmkwg\" (UniqueName: \"kubernetes.io/projected/fb187718-37b3-47fa-8c44-cd650c0185be-kube-api-access-nmkwg\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.229483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229235 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-log-socket\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.229483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229308 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dkdj\" (UniqueName: \"kubernetes.io/projected/2d2a2662-e327-4427-a3e8-482de3a8ed26-kube-api-access-4dkdj\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:24.229483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229331 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-tmp\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.229483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229431 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-crio-socket\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.229483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229467 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac082af6-5ee0-4995-9636-8271debb950a-host\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:28:24.229783 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229502 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5hs6\" (UniqueName: \"kubernetes.io/projected/14d2a2ab-d78a-47c1-8e0c-84f7418d2b81-kube-api-access-c5hs6\") pod \"migrator-66df5d496b-grlcq\" (UID: \"14d2a2ab-d78a-47c1-8e0c-84f7418d2b81\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq" Oct 31 03:28:24.229783 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229533 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:24.229783 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229740 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-tuned\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.229783 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229758 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-netns\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.229783 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229773 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-netd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.229987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229789 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.229987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229790 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-tuned\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.229987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229828 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6qd\" (UniqueName: \"kubernetes.io/projected/ce341b3b-e888-41f3-90b2-2055a4634a2d-kube-api-access-rb6qd\") pod \"ingress-canary-pr54w\" (UID: \"ce341b3b-e888-41f3-90b2-2055a4634a2d\") " pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:24.229987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229847 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-system-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.229987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229848 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e9ea6af-1755-4e15-861b-5740354972f2-original-pull-secret\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:24.229987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229878 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-etc-kubernetes\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.229987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229900 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-systemd-units\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.229987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229967 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-var-lib-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.230268 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.229992 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-ovn\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.230268 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230022 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac082af6-5ee0-4995-9636-8271debb950a-serviceca\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:28:24.230268 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230038 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:24.230268 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230065 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28de4e55-40b4-4721-9f39-2bbd1033c06e-iptables-alerter-script\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:28:24.230268 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230151 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.230268 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230186 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dd3e64a-9a74-4542-b09a-7003e749b44f-agent-certs\") pod \"konnectivity-agent-zmcjg\" (UID: \"3dd3e64a-9a74-4542-b09a-7003e749b44f\") " pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:28:24.230268 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230223 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-hosts-file\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:28:24.230268 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230252 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-netns\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230271 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-systemd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230295 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230336 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-hostroot\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230398 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-conf-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230418 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-node-log\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230439 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-run\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230461 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-lib-modules\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230484 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4db9a5-414b-4e4c-a712-e30ae75254c6-config-volume\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230502 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjqdm\" (UniqueName: \"kubernetes.io/projected/abdcefed-e204-47f4-a289-5895681e12a6-kube-api-access-jjqdm\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230519 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230573 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-multus\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.230586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230591 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230610 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovn-node-metrics-cert\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230627 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-script-lib\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230713 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-systemd\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230738 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-host\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230760 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-cnibin\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230788 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce341b3b-e888-41f3-90b2-2055a4634a2d-cert\") pod \"ingress-canary-pr54w\" (UID: \"ce341b3b-e888-41f3-90b2-2055a4634a2d\") " pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230808 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-kubelet\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230818 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230834 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2a2662-e327-4427-a3e8-482de3a8ed26-archives-path\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230864 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6fg\" (UniqueName: \"kubernetes.io/projected/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-api-access-gz6fg\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230900 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-registration-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230918 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-device-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230938 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2a2662-e327-4427-a3e8-482de3a8ed26-archives-path\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230966 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-sys-fs\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230982 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-sys\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.230998 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dd3e64a-9a74-4542-b09a-7003e749b44f-konnectivity-ca\") pod \"konnectivity-agent-zmcjg\" (UID: \"3dd3e64a-9a74-4542-b09a-7003e749b44f\") " pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:28:24.231083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231048 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231087 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-bin\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231151 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-dbus\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231172 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4db9a5-414b-4e4c-a712-e30ae75254c6-tmp-dir\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231189 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-service-ca-bundle\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231204 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgl9x\" (UniqueName: \"kubernetes.io/projected/ac082af6-5ee0-4995-9636-8271debb950a-kube-api-access-kgl9x\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231219 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-modprobe-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231233 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-kubernetes\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231276 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4db9a5-414b-4e4c-a712-e30ae75254c6-tmp-dir\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231252 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-binary-copy\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231307 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq8vh\" (UniqueName: \"kubernetes.io/projected/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-kube-api-access-dq8vh\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231329 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-config\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231358 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4jzx\" (UniqueName: \"kubernetes.io/projected/ad4db9a5-414b-4e4c-a712-e30ae75254c6-kube-api-access-w4jzx\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231384 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/abdcefed-e204-47f4-a289-5895681e12a6-signing-key\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231406 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/abdcefed-e204-47f4-a289-5895681e12a6-signing-cabundle\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231423 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-multus-certs\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231441 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-slash\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231460 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231556 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-conf\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231581 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdfsd\" (UniqueName: \"kubernetes.io/projected/28de4e55-40b4-4721-9f39-2bbd1033c06e-kube-api-access-zdfsd\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:28:24.231650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231609 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-socket-dir-parent\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.232388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231676 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-kubelet\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.232388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231707 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-etc-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.232388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231737 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-env-overrides\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.232388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231765 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-trusted-ca-bundle\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:24.232388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231796 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.232388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231826 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76gl\" (UniqueName: \"kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl\") pod \"network-check-target-bh9xh\" (UID: \"fc1bac22-a12b-4777-989e-a4c059633fd1\") " pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:24.232388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231854 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.232388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231880 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cnibin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.232388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231906 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cni-binary-copy\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.232388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.231937 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpjn\" (UniqueName: \"kubernetes.io/projected/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kube-api-access-jkpjn\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.289303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.289273 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckh87\" (UniqueName: \"kubernetes.io/projected/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-kube-api-access-ckh87\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:24.294482 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.294462 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtjc\" (UniqueName: \"kubernetes.io/projected/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-kube-api-access-hrtjc\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.294482 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.294477 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmkwg\" (UniqueName: \"kubernetes.io/projected/fb187718-37b3-47fa-8c44-cd650c0185be-kube-api-access-nmkwg\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.329868 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.329797 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 03:28:24.330227 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.330208 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 03:28:24.330355 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.330339 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 03:28:24.330404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.330388 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 03:28:24.330464 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.330391 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 03:28:24.330684 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.330666 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.330737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.330684 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 03:28:24.330737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.330670 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 03:28:24.330737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.330703 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 03:28:24.330859 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.330830 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovn-node-metrics-cert\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.330993 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.330977 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce341b3b-e888-41f3-90b2-2055a4634a2d-cert\") pod \"ingress-canary-pr54w\" (UID: \"ce341b3b-e888-41f3-90b2-2055a4634a2d\") " pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:24.331066 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331052 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8tvwd\"" Oct 31 03:28:24.331100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331060 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.331298 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331279 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.331357 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331341 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 03:28:24.331357 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331344 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.331446 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331373 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hp8t5\"" Oct 31 03:28:24.331446 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331407 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-service-ca-bundle\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:24.331446 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331414 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dd3e64a-9a74-4542-b09a-7003e749b44f-konnectivity-ca\") pod \"konnectivity-agent-zmcjg\" (UID: \"3dd3e64a-9a74-4542-b09a-7003e749b44f\") " pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:28:24.331644 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331626 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.331732 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331693 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.331732 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331716 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.331846 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331827 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bzhcg\"" Oct 31 03:28:24.331907 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331863 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.331907 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331867 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.331907 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331871 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dvnpq\"" Oct 31 03:28:24.331907 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331883 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 03:28:24.332096 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331928 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fs258\"" Oct 31 03:28:24.332096 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331965 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q48mz\"" Oct 31 03:28:24.332096 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332066 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.332288 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332203 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cnibin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.332288 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332224 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28de4e55-40b4-4721-9f39-2bbd1033c06e-host-slash\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:28:24.332288 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332242 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-bin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.332288 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332260 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-var-lib-kubelet\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.332511 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.331871 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/abdcefed-e204-47f4-a289-5895681e12a6-signing-key\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:24.332511 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332397 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cnibin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.332511 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332433 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28de4e55-40b4-4721-9f39-2bbd1033c06e-host-slash\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:28:24.332511 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332465 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-bin\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.332704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332534 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2rc66\"" Oct 31 03:28:24.332704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332567 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.332704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332589 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.332704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332615 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 03:28:24.332704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332633 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.332704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332641 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.332704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332684 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.332704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332686 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 03:28:24.332704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332695 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.332704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332668 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-k8s-cni-cncf-io\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332722 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jdz67\"" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332725 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332734 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332739 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-k8s-cni-cncf-io\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332753 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zvcgd\"" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332741 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-var-lib-kubelet\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332770 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332799 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysconfig\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332819 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332825 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.332841 4426 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332856 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-system-cni-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332867 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysconfig\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332885 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.332919 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:24.832895336 +0000 UTC m=+1.833520559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : secret "insights-runtime-extractor-tls" not found Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332924 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-system-cni-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332939 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-socket-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332964 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-kubelet-config\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.332983 4426 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.332994 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-log-socket\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333000 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333021 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-crio-socket\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.333050 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls podName:ad4db9a5-414b-4e4c-a712-e30ae75254c6 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:24.833040084 +0000 UTC m=+1.833665293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls") pod "dns-default-5rfb8" (UID: "ad4db9a5-414b-4e4c-a712-e30ae75254c6") : secret "dns-default-metrics-tls" not found Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333047 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-log-socket\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333060 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-kubelet-config\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:24.333048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333002 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-socket-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333064 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac082af6-5ee0-4995-9636-8271debb950a-host\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333080 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-crio-socket\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333088 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac082af6-5ee0-4995-9636-8271debb950a-host\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333148 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-netns\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333171 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-netd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333187 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333228 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333232 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-netd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333264 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-system-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333287 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-etc-kubernetes\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333303 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-systemd-units\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333265 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-netns\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333318 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-var-lib-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333335 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-ovn\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333343 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-etc-kubernetes\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333374 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-var-lib-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333377 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-system-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333391 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333377 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-systemd-units\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333392 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-ovn\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333421 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333441 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-hosts-file\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333457 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-netns\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.333467 4426 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333491 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-systemd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333541 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-netns\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333471 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-systemd\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333541 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-hosts-file\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.333559 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:24.833543168 +0000 UTC m=+1.834168389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : configmap "kube-rbac-proxy" not found Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333589 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333605 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-hostroot\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333643 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333644 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-hostroot\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333674 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-conf-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333689 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-node-log\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333705 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-run\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333722 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-lib-modules\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333720 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-conf-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333742 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-node-log\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333761 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-run\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333761 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjqdm\" (UniqueName: \"kubernetes.io/projected/abdcefed-e204-47f4-a289-5895681e12a6-kube-api-access-jjqdm\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333801 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-multus\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333814 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-lib-modules\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.333807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333824 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333848 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-systemd\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333854 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-cni-multus\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333879 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-host\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333906 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-cnibin\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333918 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-run-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333931 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-systemd\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333933 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-host\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333942 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-kubelet\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333943 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb187718-37b3-47fa-8c44-cd650c0185be-cnibin\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333973 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-registration-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.333991 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-device-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334024 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-sys-fs\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334033 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-var-lib-kubelet\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334046 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-device-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334042 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-sys\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334045 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-registration-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334090 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-sys-fs\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334066 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-sys\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334141 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334186 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-bin\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334209 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-dbus\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334227 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-cni-dir\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334235 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-modprobe-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334235 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-cni-bin\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334262 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-kubernetes\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334305 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-modprobe-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334299 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e9ea6af-1755-4e15-861b-5740354972f2-dbus\") pod \"global-pull-secret-syncer-n9rx6\" (UID: \"6e9ea6af-1755-4e15-861b-5740354972f2\") " pod="kube-system/global-pull-secret-syncer-n9rx6" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334365 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-kubernetes\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334363 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-multus-certs\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334436 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-slash\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334456 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-host-run-multus-certs\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334464 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334496 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-slash\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334493 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-conf\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334549 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-socket-dir-parent\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334562 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4jzx\" (UniqueName: \"kubernetes.io/projected/ad4db9a5-414b-4e4c-a712-e30ae75254c6-kube-api-access-w4jzx\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334573 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-d\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334578 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-kubelet\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334629 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-host-kubelet\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334632 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-multus-socket-dir-parent\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334633 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7bf98c9-3458-4936-8dfd-32faa8e522b9-etc-sysctl-conf\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334657 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-etc-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334689 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334700 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdfsd\" (UniqueName: \"kubernetes.io/projected/28de4e55-40b4-4721-9f39-2bbd1033c06e-kube-api-access-zdfsd\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334706 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/288945de-f0e7-47c7-87c9-45f8dbc021d9-etc-openvswitch\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334761 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.334810 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgl9x\" (UniqueName: \"kubernetes.io/projected/ac082af6-5ee0-4995-9636-8271debb950a-kube-api-access-kgl9x\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335084 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335231 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335244 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335282 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvwkt\"" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335318 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hp5ds\"" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335344 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 03:28:24.335356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335387 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z9sxx\"" Oct 31 03:28:24.336893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335409 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lj989\"" Oct 31 03:28:24.336893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335417 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.336893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335417 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8nlcv\"" Oct 31 03:28:24.336893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335415 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 03:28:24.336893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335466 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 03:28:24.336893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335538 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.336893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335555 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 03:28:24.336893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.335569 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:28:24.340508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340323 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5hs6\" (UniqueName: \"kubernetes.io/projected/14d2a2ab-d78a-47c1-8e0c-84f7418d2b81-kube-api-access-c5hs6\") pod \"migrator-66df5d496b-grlcq\" (UID: \"14d2a2ab-d78a-47c1-8e0c-84f7418d2b81\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq" Oct 31 03:28:24.340508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340336 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6fg\" (UniqueName: \"kubernetes.io/projected/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-api-access-gz6fg\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.340508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340350 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqlc\" (UniqueName: \"kubernetes.io/projected/f7bf98c9-3458-4936-8dfd-32faa8e522b9-kube-api-access-mdqlc\") pod \"tuned-v5d8b\" (UID: \"f7bf98c9-3458-4936-8dfd-32faa8e522b9\") " pod="openshift-cluster-node-tuning-operator/tuned-v5d8b" Oct 31 03:28:24.340508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340369 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpjn\" (UniqueName: \"kubernetes.io/projected/d7c37d3a-09c1-48a2-bd7b-abf3975ca24e-kube-api-access-jkpjn\") pod \"aws-ebs-csi-driver-node-pd428\" (UID: \"d7c37d3a-09c1-48a2-bd7b-abf3975ca24e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pd428" Oct 31 03:28:24.340508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340408 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcs67\" (UniqueName: \"kubernetes.io/projected/288945de-f0e7-47c7-87c9-45f8dbc021d9-kube-api-access-xcs67\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.340508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340443 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq8vh\" (UniqueName: \"kubernetes.io/projected/bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49-kube-api-access-dq8vh\") pod \"node-resolver-2vww2\" (UID: \"bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49\") " pod="openshift-dns/node-resolver-2vww2" Oct 31 03:28:24.340508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340468 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 03:28:24.340877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340554 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac082af6-5ee0-4995-9636-8271debb950a-serviceca\") pod \"node-ca-fb4pz\" (UID: \"ac082af6-5ee0-4995-9636-8271debb950a\") " pod="openshift-image-registry/node-ca-fb4pz" Oct 31 03:28:24.340877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340593 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dd3e64a-9a74-4542-b09a-7003e749b44f-agent-certs\") pod \"konnectivity-agent-zmcjg\" (UID: \"3dd3e64a-9a74-4542-b09a-7003e749b44f\") " pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:28:24.340877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340597 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76gl\" (UniqueName: \"kubernetes.io/projected/fc1bac22-a12b-4777-989e-a4c059633fd1-kube-api-access-p76gl\") pod \"network-check-target-bh9xh\" (UID: \"fc1bac22-a12b-4777-989e-a4c059633fd1\") " pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:24.340877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340621 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dkdj\" (UniqueName: \"kubernetes.io/projected/2d2a2662-e327-4427-a3e8-482de3a8ed26-kube-api-access-4dkdj\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:24.340877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340625 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28de4e55-40b4-4721-9f39-2bbd1033c06e-iptables-alerter-script\") pod \"iptables-alerter-lt92b\" (UID: \"28de4e55-40b4-4721-9f39-2bbd1033c06e\") " pod="openshift-network-operator/iptables-alerter-lt92b" Oct 31 03:28:24.340877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340713 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4db9a5-414b-4e4c-a712-e30ae75254c6-config-volume\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:24.341076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.340922 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-script-lib\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.341924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.341902 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-env-overrides\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.341924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.341916 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/abdcefed-e204-47f4-a289-5895681e12a6-signing-cabundle\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:24.342010 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.341955 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-binary-copy\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.342010 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.341958 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/288945de-f0e7-47c7-87c9-45f8dbc021d9-ovnkube-config\") pod \"ovnkube-node-4hh72\" (UID: \"288945de-f0e7-47c7-87c9-45f8dbc021d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:24.342086 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.342034 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb187718-37b3-47fa-8c44-cd650c0185be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7txzf\" (UID: \"fb187718-37b3-47fa-8c44-cd650c0185be\") " pod="openshift-multus/multus-additional-cni-plugins-7txzf" Oct 31 03:28:24.342086 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.342071 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0-cni-binary-copy\") pod \"multus-9sx4r\" (UID: \"2f2dbb0c-6baf-43d3-8fa9-3534f9fe8eb0\") " pod="openshift-multus/multus-9sx4r" Oct 31 03:28:24.345587 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.345570 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6qd\" (UniqueName: \"kubernetes.io/projected/ce341b3b-e888-41f3-90b2-2055a4634a2d-kube-api-access-rb6qd\") pod \"ingress-canary-pr54w\" (UID: \"ce341b3b-e888-41f3-90b2-2055a4634a2d\") " pod="openshift-ingress-canary/ingress-canary-pr54w" Oct 31 03:28:24.377645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.377449 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:28:24.379879 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.378316 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 03:28:24.382219 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.382199 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-trusted-ca-bundle\") pod \"periodic-gathering-hnt22-469x9\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:28:24.384174 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.384159 4426 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 03:28:24.384245 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.384232 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs podName:ffc3ecb5-d1e9-4125-a700-7c4b992b75bf nodeName:}" failed. No retries permitted until 2026-10-31 03:28:24.884213104 +0000 UTC m=+1.884838326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs") pod "network-metrics-daemon-rh4rl" (UID: "ffc3ecb5-d1e9-4125-a700-7c4b992b75bf") : secret "metrics-daemon-secret" not found Oct 31 03:28:24.390278 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.390257 4426 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 03:28:24.392523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.392499 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjqdm\" (UniqueName: \"kubernetes.io/projected/abdcefed-e204-47f4-a289-5895681e12a6-kube-api-access-jjqdm\") pod \"service-ca-5fffc97bdf-sjcw2\" (UID: \"abdcefed-e204-47f4-a289-5895681e12a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:24.481314 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.481292 4426 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:28:24.481481 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.481435 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" Oct 31 03:28:24.838700 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.838665 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.838885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.838705 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:24.838885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.838744 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:24.838885 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.838825 4426 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:28:24.838885 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.838829 4426 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:28:24.838885 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.838877 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:25.838860578 +0000 UTC m=+2.839485784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : configmap "kube-rbac-proxy" not found Oct 31 03:28:24.838885 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.838826 4426 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:28:24.839114 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.838905 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:25.8388875 +0000 UTC m=+2.839512707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : secret "insights-runtime-extractor-tls" not found Oct 31 03:28:24.839114 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.838933 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls podName:ad4db9a5-414b-4e4c-a712-e30ae75254c6 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:25.838922366 +0000 UTC m=+2.839547573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls") pod "dns-default-5rfb8" (UID: "ad4db9a5-414b-4e4c-a712-e30ae75254c6") : secret "dns-default-metrics-tls" not found Oct 31 03:28:24.873890 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.873853 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sjcw2"] Oct 31 03:28:24.878301 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:24.878268 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdcefed_e204_47f4_a289_5895681e12a6.slice/crio-cdbb8da42200ce47675b2864e9eed5e411653cf4a972fbfc3f9b6133f5d385c3 WatchSource:0}: Error finding container cdbb8da42200ce47675b2864e9eed5e411653cf4a972fbfc3f9b6133f5d385c3: Status 404 returned error can't find the container with id cdbb8da42200ce47675b2864e9eed5e411653cf4a972fbfc3f9b6133f5d385c3 Oct 31 03:28:24.939263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:24.939231 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:24.939443 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.939391 4426 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 03:28:24.939486 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:24.939454 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs podName:ffc3ecb5-d1e9-4125-a700-7c4b992b75bf nodeName:}" failed. No retries permitted until 2026-10-31 03:28:25.939440414 +0000 UTC m=+2.940065619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs") pod "network-metrics-daemon-rh4rl" (UID: "ffc3ecb5-d1e9-4125-a700-7c4b992b75bf") : secret "metrics-daemon-secret" not found Oct 31 03:28:25.162954 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:25.162923 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" event={"ID":"abdcefed-e204-47f4-a289-5895681e12a6","Type":"ContainerStarted","Data":"cdbb8da42200ce47675b2864e9eed5e411653cf4a972fbfc3f9b6133f5d385c3"} Oct 31 03:28:25.847244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:25.847166 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:25.847244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:25.847210 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:25.847244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:25.847244 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:25.847499 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:25.847348 4426 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:28:25.847499 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:25.847382 4426 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:28:25.847499 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:25.847429 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:27.847404086 +0000 UTC m=+4.848029308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : secret "insights-runtime-extractor-tls" not found Oct 31 03:28:25.847499 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:25.847461 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls podName:ad4db9a5-414b-4e4c-a712-e30ae75254c6 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:27.847451596 +0000 UTC m=+4.848076808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls") pod "dns-default-5rfb8" (UID: "ad4db9a5-414b-4e4c-a712-e30ae75254c6") : secret "dns-default-metrics-tls" not found Oct 31 03:28:25.847836 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:25.847821 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:25.947937 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:25.947868 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:25.948084 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:25.948038 4426 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 03:28:25.948159 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:25.948095 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs podName:ffc3ecb5-d1e9-4125-a700-7c4b992b75bf nodeName:}" failed. No retries permitted until 2026-10-31 03:28:27.948081319 +0000 UTC m=+4.948706539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs") pod "network-metrics-daemon-rh4rl" (UID: "ffc3ecb5-d1e9-4125-a700-7c4b992b75bf") : secret "metrics-daemon-secret" not found Oct 31 03:28:26.703091 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:26.703060 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:27.170892 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:27.170852 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-grlcq" event={"ID":"14d2a2ab-d78a-47c1-8e0c-84f7418d2b81","Type":"ContainerStarted","Data":"3768a7a8a0d1994438766cba8f5daa6629836ba2e542afe463b7452dbdacd44f"} Oct 31 03:28:27.846681 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:27.846652 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zmcjg" Oct 31 03:28:27.865370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:27.865341 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:27.865467 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:27.865385 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:27.865525 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:27.865487 4426 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:28:27.865559 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:27.865524 4426 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:28:27.865559 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:27.865552 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls podName:da393ceb-f1dc-4cb1-b82f-933a8eaa8497 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:31.865536376 +0000 UTC m=+8.866161586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lhsn4" (UID: "da393ceb-f1dc-4cb1-b82f-933a8eaa8497") : secret "insights-runtime-extractor-tls" not found Oct 31 03:28:27.865621 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:27.865572 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls podName:ad4db9a5-414b-4e4c-a712-e30ae75254c6 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:31.865558462 +0000 UTC m=+8.866183670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls") pod "dns-default-5rfb8" (UID: "ad4db9a5-414b-4e4c-a712-e30ae75254c6") : secret "dns-default-metrics-tls" not found Oct 31 03:28:27.966152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:27.966051 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:27.966305 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:27.966210 4426 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 03:28:27.966305 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:27.966282 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs podName:ffc3ecb5-d1e9-4125-a700-7c4b992b75bf nodeName:}" failed. No retries permitted until 2026-10-31 03:28:31.96626752 +0000 UTC m=+8.966892746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs") pod "network-metrics-daemon-rh4rl" (UID: "ffc3ecb5-d1e9-4125-a700-7c4b992b75bf") : secret "metrics-daemon-secret" not found Oct 31 03:28:28.082503 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:28.082471 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:28.082756 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:28.082667 4426 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:28:28.174987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:28.174955 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sjcw2" event={"ID":"abdcefed-e204-47f4-a289-5895681e12a6","Type":"ContainerStarted","Data":"83919d15e071a2d33dd3e24c0b3a01c71d8a0392956fce16b53395353df0ff71"} Oct 31 03:28:28.176558 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:28.176533 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hnt22-469x9" event={"ID":"2d2a2662-e327-4427-a3e8-482de3a8ed26","Type":"ContainerStarted","Data":"7861833dddccd59bc60270491bde6231533a30990a5d10874928513dd24f7490"} Oct 31 03:28:28.199490 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:28.199468 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:31.126223 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:31.126193 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4hh72" Oct 31 03:28:31.893119 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:31.893079 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:31.893310 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:31.893143 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:31.893310 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:31.893277 4426 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:28:31.893398 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:28:31.893343 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls podName:ad4db9a5-414b-4e4c-a712-e30ae75254c6 nodeName:}" failed. No retries permitted until 2026-10-31 03:28:39.89332538 +0000 UTC m=+16.893950589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls") pod "dns-default-5rfb8" (UID: "ad4db9a5-414b-4e4c-a712-e30ae75254c6") : secret "dns-default-metrics-tls" not found Oct 31 03:28:31.895348 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:31.895318 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da393ceb-f1dc-4cb1-b82f-933a8eaa8497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lhsn4\" (UID: \"da393ceb-f1dc-4cb1-b82f-933a8eaa8497\") " pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:31.981669 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:31.981636 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lhsn4" Oct 31 03:28:31.994007 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:31.993977 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:31.996482 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:31.996461 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3ecb5-d1e9-4125-a700-7c4b992b75bf-metrics-certs\") pod \"network-metrics-daemon-rh4rl\" (UID: \"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf\") " pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:32.276187 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:32.276096 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4rl" Oct 31 03:28:32.344034 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:32.344004 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lhsn4"] Oct 31 03:28:32.348712 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:32.348688 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda393ceb_f1dc_4cb1_b82f_933a8eaa8497.slice/crio-eb1a10d9c0c2c594de906cb0b54a69435f8c740e626a02eda1ca781074518558 WatchSource:0}: Error finding container eb1a10d9c0c2c594de906cb0b54a69435f8c740e626a02eda1ca781074518558: Status 404 returned error can't find the container with id eb1a10d9c0c2c594de906cb0b54a69435f8c740e626a02eda1ca781074518558 Oct 31 03:28:32.637596 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:32.637527 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rh4rl"] Oct 31 03:28:32.641385 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:32.641347 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc3ecb5_d1e9_4125_a700_7c4b992b75bf.slice/crio-8fef7ee21b10cb2bd938771200378b784fdc529c54752aed885fecc0152387a2 WatchSource:0}: Error finding container 8fef7ee21b10cb2bd938771200378b784fdc529c54752aed885fecc0152387a2: Status 404 returned error can't find the container with id 8fef7ee21b10cb2bd938771200378b784fdc529c54752aed885fecc0152387a2 Oct 31 03:28:32.732526 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:32.732498 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2vww2_bbb1e064-f4aa-4576-ae1e-a7bc8ffa7d49/dns-node-resolver/0.log" Oct 31 03:28:33.189538 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:33.189491 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh4rl" event={"ID":"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf","Type":"ContainerStarted","Data":"8fef7ee21b10cb2bd938771200378b784fdc529c54752aed885fecc0152387a2"} Oct 31 03:28:33.190854 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:33.190833 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lhsn4" event={"ID":"da393ceb-f1dc-4cb1-b82f-933a8eaa8497","Type":"ContainerStarted","Data":"398b704ef8559f35e6a6fddf9ca5ebbfb309c95d9fe1aaf0dd0bafcb1302a561"} Oct 31 03:28:33.190910 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:33.190877 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lhsn4" event={"ID":"da393ceb-f1dc-4cb1-b82f-933a8eaa8497","Type":"ContainerStarted","Data":"eb1a10d9c0c2c594de906cb0b54a69435f8c740e626a02eda1ca781074518558"} Oct 31 03:28:33.933057 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:33.933022 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fb4pz_ac082af6-5ee0-4995-9636-8271debb950a/node-ca/0.log" Oct 31 03:28:34.196732 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:34.196698 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lhsn4" event={"ID":"da393ceb-f1dc-4cb1-b82f-933a8eaa8497","Type":"ContainerStarted","Data":"3076013d67ad47fbdab3e67b6dbd9db75c64d6db7a269da939648aa9d31d5cf7"} Oct 31 03:28:34.733531 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:34.733500 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pr54w_ce341b3b-e888-41f3-90b2-2055a4634a2d/serve-healthcheck-canary/0.log" Oct 31 03:28:34.933512 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:34.933413 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-grlcq_14d2a2ab-d78a-47c1-8e0c-84f7418d2b81/migrator/0.log" Oct 31 03:28:35.203036 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:35.202957 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh4rl" event={"ID":"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf","Type":"ContainerStarted","Data":"6c6802fedad04095be1836dc00242c0c0d0c76b5fd815514ec0f956f795aea14"} Oct 31 03:28:35.203036 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:35.203009 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh4rl" event={"ID":"ffc3ecb5-d1e9-4125-a700-7c4b992b75bf","Type":"ContainerStarted","Data":"87584504e2f3e12992c45f1d4928e7a448490beea1e82ab28eabded256a2ebcd"} Oct 31 03:28:36.208473 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:36.208425 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lhsn4" event={"ID":"da393ceb-f1dc-4cb1-b82f-933a8eaa8497","Type":"ContainerStarted","Data":"4e25f1f8e2a7da535a55c473c2af7e59b7ede570fc3275d8e4ba4347ba45905c"} Oct 31 03:28:39.959976 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:39.959937 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:39.962224 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:39.962202 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad4db9a5-414b-4e4c-a712-e30ae75254c6-metrics-tls\") pod \"dns-default-5rfb8\" (UID: \"ad4db9a5-414b-4e4c-a712-e30ae75254c6\") " pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:40.080448 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:40.080409 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:40.440748 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:40.440711 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5rfb8"] Oct 31 03:28:40.444482 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:40.444455 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4db9a5_414b_4e4c_a712_e30ae75254c6.slice/crio-3a645462d3ab4120a17364898314f29506a433fe6a327ca53572cd5d276e0b87 WatchSource:0}: Error finding container 3a645462d3ab4120a17364898314f29506a433fe6a327ca53572cd5d276e0b87: Status 404 returned error can't find the container with id 3a645462d3ab4120a17364898314f29506a433fe6a327ca53572cd5d276e0b87 Oct 31 03:28:41.224914 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:41.224876 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rfb8" event={"ID":"ad4db9a5-414b-4e4c-a712-e30ae75254c6","Type":"ContainerStarted","Data":"3a645462d3ab4120a17364898314f29506a433fe6a327ca53572cd5d276e0b87"} Oct 31 03:28:43.234057 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.234012 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rfb8" event={"ID":"ad4db9a5-414b-4e4c-a712-e30ae75254c6","Type":"ContainerStarted","Data":"1f79c1f688eb35a54d1f9727622f6296ba11326699b675b5905adaa2070f2690"} Oct 31 03:28:43.234443 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.234086 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rfb8" event={"ID":"ad4db9a5-414b-4e4c-a712-e30ae75254c6","Type":"ContainerStarted","Data":"a480c8f2eaa8edad4af2b132472f03f1db382b4ce5ad1c3e53dbc38df415a71f"} Oct 31 03:28:43.234443 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.234156 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:43.699004 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.698964 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7"] Oct 31 03:28:43.701824 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.701803 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-j9lnw"] Oct 31 03:28:43.701950 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.701928 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7" Oct 31 03:28:43.704712 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.704694 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j9lnw" Oct 31 03:28:43.753003 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.752962 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 03:28:43.753223 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.752962 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 03:28:43.787590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.787562 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7216fab-cc51-42e9-990a-3690b21846f5-tmp\") pod \"downloads-6dd789c8b7-j9lnw\" (UID: \"c7216fab-cc51-42e9-990a-3690b21846f5\") " pod="openshift-console/downloads-6dd789c8b7-j9lnw" Oct 31 03:28:43.787590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.787594 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4d6bf979-c514-43c6-a329-98ae2336711d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kmdh7\" (UID: \"4d6bf979-c514-43c6-a329-98ae2336711d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7" Oct 31 03:28:43.787790 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.787618 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dnk\" (UniqueName: \"kubernetes.io/projected/c7216fab-cc51-42e9-990a-3690b21846f5-kube-api-access-d7dnk\") pod \"downloads-6dd789c8b7-j9lnw\" (UID: \"c7216fab-cc51-42e9-990a-3690b21846f5\") " pod="openshift-console/downloads-6dd789c8b7-j9lnw" Oct 31 03:28:43.799135 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.799092 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-sc22r\"" Oct 31 03:28:43.799135 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.799104 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6ndvp\"" Oct 31 03:28:43.799135 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.799092 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 03:28:43.799901 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.799886 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-676475bf7d-q7pvn"] Oct 31 03:28:43.802861 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.802846 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.817747 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.817722 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j9lnw"] Oct 31 03:28:43.853856 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.853833 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 03:28:43.854250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.854233 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 03:28:43.856770 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.856750 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xg7xf\"" Oct 31 03:28:43.856770 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.856755 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 03:28:43.888118 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.888085 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dnk\" (UniqueName: \"kubernetes.io/projected/c7216fab-cc51-42e9-990a-3690b21846f5-kube-api-access-d7dnk\") pod \"downloads-6dd789c8b7-j9lnw\" (UID: \"c7216fab-cc51-42e9-990a-3690b21846f5\") " pod="openshift-console/downloads-6dd789c8b7-j9lnw" Oct 31 03:28:43.888290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.888160 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd2c7fae-1771-43c0-8353-111068097213-installation-pull-secrets\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.888290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.888192 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd2c7fae-1771-43c0-8353-111068097213-bound-sa-token\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.888290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.888235 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd2c7fae-1771-43c0-8353-111068097213-trusted-ca\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.888455 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.888293 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd2c7fae-1771-43c0-8353-111068097213-ca-trust-extracted\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.888455 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.888407 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7216fab-cc51-42e9-990a-3690b21846f5-tmp\") pod \"downloads-6dd789c8b7-j9lnw\" (UID: \"c7216fab-cc51-42e9-990a-3690b21846f5\") " pod="openshift-console/downloads-6dd789c8b7-j9lnw" Oct 31 03:28:43.888455 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.888438 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4d6bf979-c514-43c6-a329-98ae2336711d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kmdh7\" (UID: \"4d6bf979-c514-43c6-a329-98ae2336711d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7" Oct 31 03:28:43.888607 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.888470 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2c7fae-1771-43c0-8353-111068097213-registry-tls\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.888607 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.888500 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzp6c\" (UniqueName: \"kubernetes.io/projected/bd2c7fae-1771-43c0-8353-111068097213-kube-api-access-tzp6c\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.888607 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.888547 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd2c7fae-1771-43c0-8353-111068097213-image-registry-private-configuration\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.888728 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.888615 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd2c7fae-1771-43c0-8353-111068097213-registry-certificates\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.889055 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.889035 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7216fab-cc51-42e9-990a-3690b21846f5-tmp\") pod \"downloads-6dd789c8b7-j9lnw\" (UID: \"c7216fab-cc51-42e9-990a-3690b21846f5\") " pod="openshift-console/downloads-6dd789c8b7-j9lnw" Oct 31 03:28:43.891338 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.891314 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4d6bf979-c514-43c6-a329-98ae2336711d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kmdh7\" (UID: \"4d6bf979-c514-43c6-a329-98ae2336711d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7" Oct 31 03:28:43.900460 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.900432 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 03:28:43.901213 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.901196 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-676475bf7d-q7pvn"] Oct 31 03:28:43.901718 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.901701 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7"] Oct 31 03:28:43.948233 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.948210 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dnk\" (UniqueName: \"kubernetes.io/projected/c7216fab-cc51-42e9-990a-3690b21846f5-kube-api-access-d7dnk\") pod \"downloads-6dd789c8b7-j9lnw\" (UID: \"c7216fab-cc51-42e9-990a-3690b21846f5\") " pod="openshift-console/downloads-6dd789c8b7-j9lnw" Oct 31 03:28:43.989007 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.988943 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzp6c\" (UniqueName: \"kubernetes.io/projected/bd2c7fae-1771-43c0-8353-111068097213-kube-api-access-tzp6c\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.989007 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.988988 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd2c7fae-1771-43c0-8353-111068097213-image-registry-private-configuration\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.989171 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.989103 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd2c7fae-1771-43c0-8353-111068097213-registry-certificates\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.989213 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.989178 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd2c7fae-1771-43c0-8353-111068097213-installation-pull-secrets\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.989213 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.989198 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd2c7fae-1771-43c0-8353-111068097213-bound-sa-token\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.989325 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.989230 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd2c7fae-1771-43c0-8353-111068097213-trusted-ca\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.989325 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.989250 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd2c7fae-1771-43c0-8353-111068097213-ca-trust-extracted\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.989452 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.989399 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2c7fae-1771-43c0-8353-111068097213-registry-tls\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.989674 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.989650 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd2c7fae-1771-43c0-8353-111068097213-ca-trust-extracted\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.990063 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.990038 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd2c7fae-1771-43c0-8353-111068097213-registry-certificates\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.990378 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.990359 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd2c7fae-1771-43c0-8353-111068097213-trusted-ca\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.991401 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.991375 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd2c7fae-1771-43c0-8353-111068097213-image-registry-private-configuration\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.991544 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.991525 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd2c7fae-1771-43c0-8353-111068097213-installation-pull-secrets\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:43.991678 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:43.991661 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2c7fae-1771-43c0-8353-111068097213-registry-tls\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:44.015573 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:44.015551 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7" Oct 31 03:28:44.020643 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:44.020619 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j9lnw" Oct 31 03:28:44.046357 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:44.046331 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzp6c\" (UniqueName: \"kubernetes.io/projected/bd2c7fae-1771-43c0-8353-111068097213-kube-api-access-tzp6c\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:44.046604 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:44.046572 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd2c7fae-1771-43c0-8353-111068097213-bound-sa-token\") pod \"image-registry-676475bf7d-q7pvn\" (UID: \"bd2c7fae-1771-43c0-8353-111068097213\") " pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:44.113913 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:44.113882 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:44.377674 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:44.377597 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7"] Oct 31 03:28:44.381012 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:44.380981 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6bf979_c514_43c6_a329_98ae2336711d.slice/crio-77faff8a6a513df4e5c03d22c238d06ead967d4e6de070471c30c6341405a488 WatchSource:0}: Error finding container 77faff8a6a513df4e5c03d22c238d06ead967d4e6de070471c30c6341405a488: Status 404 returned error can't find the container with id 77faff8a6a513df4e5c03d22c238d06ead967d4e6de070471c30c6341405a488 Oct 31 03:28:44.426405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:44.426328 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j9lnw"] Oct 31 03:28:44.429800 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:44.429773 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7216fab_cc51_42e9_990a_3690b21846f5.slice/crio-f6c6290e1ac7cf665ebcbfcb99f05c63925d07d08924b0a4bd9d6ebd3771cdb5 WatchSource:0}: Error finding container f6c6290e1ac7cf665ebcbfcb99f05c63925d07d08924b0a4bd9d6ebd3771cdb5: Status 404 returned error can't find the container with id f6c6290e1ac7cf665ebcbfcb99f05c63925d07d08924b0a4bd9d6ebd3771cdb5 Oct 31 03:28:44.478239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:44.478216 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-676475bf7d-q7pvn"] Oct 31 03:28:44.481628 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:28:44.481597 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2c7fae_1771_43c0_8353_111068097213.slice/crio-14452f736fc466b8d1bee7458b27fd99ce8f41a2f309f62afc91d13dc0d14128 WatchSource:0}: Error finding container 14452f736fc466b8d1bee7458b27fd99ce8f41a2f309f62afc91d13dc0d14128: Status 404 returned error can't find the container with id 14452f736fc466b8d1bee7458b27fd99ce8f41a2f309f62afc91d13dc0d14128 Oct 31 03:28:45.241406 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:45.241363 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j9lnw" event={"ID":"c7216fab-cc51-42e9-990a-3690b21846f5","Type":"ContainerStarted","Data":"f6c6290e1ac7cf665ebcbfcb99f05c63925d07d08924b0a4bd9d6ebd3771cdb5"} Oct 31 03:28:45.242628 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:45.242601 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7" event={"ID":"4d6bf979-c514-43c6-a329-98ae2336711d","Type":"ContainerStarted","Data":"77faff8a6a513df4e5c03d22c238d06ead967d4e6de070471c30c6341405a488"} Oct 31 03:28:45.243983 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:45.243957 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" event={"ID":"bd2c7fae-1771-43c0-8353-111068097213","Type":"ContainerStarted","Data":"4810d6f45603d729241c1de00d84ea6b625e917e28ef1e00f5645a2b3d23f188"} Oct 31 03:28:45.244073 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:45.244003 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" event={"ID":"bd2c7fae-1771-43c0-8353-111068097213","Type":"ContainerStarted","Data":"14452f736fc466b8d1bee7458b27fd99ce8f41a2f309f62afc91d13dc0d14128"} Oct 31 03:28:45.244117 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:45.244101 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:28:45.360262 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:45.360212 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" podStartSLOduration=2.360198367 podStartE2EDuration="2.360198367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:45.359402897 +0000 UTC m=+22.360028124" watchObservedRunningTime="2026-10-31 03:28:45.360198367 +0000 UTC m=+22.360823585" Oct 31 03:28:47.253564 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:47.253516 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7" event={"ID":"4d6bf979-c514-43c6-a329-98ae2336711d","Type":"ContainerStarted","Data":"d5543dbb1596c70e5ea7b762ff59b87835d8cd4595e121b7bfca0dea357bd84c"} Oct 31 03:28:47.254021 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:47.253995 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7" Oct 31 03:28:47.259104 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:47.259079 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7" Oct 31 03:28:47.362396 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:47.362338 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kmdh7" podStartSLOduration=2.424721158 podStartE2EDuration="4.362317836s" totalImagesPullingTime="1.937596678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:47.361524587 +0000 UTC m=+24.362149816" watchObservedRunningTime="2026-10-31 03:28:47.362317836 +0000 UTC m=+24.362943058" Oct 31 03:28:53.240097 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:53.240062 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5rfb8" Oct 31 03:28:55.574539 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.574506 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wbwhr"] Oct 31 03:28:55.593800 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.593768 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.756826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.756782 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 03:28:55.756826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.756809 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 03:28:55.757085 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.756942 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 03:28:55.757085 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.756992 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 03:28:55.757085 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.757008 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b9rvf\"" Oct 31 03:28:55.757085 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.757074 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 03:28:55.758080 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.758061 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 03:28:55.781059 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.781033 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-root\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.781059 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.781072 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-textfile\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.781286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.781092 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.781286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.781113 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-sys\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.781286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.781178 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-tls\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.781286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.781230 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.781286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.781279 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-metrics-client-ca\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.781493 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.781321 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-wtmp\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.781493 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.781352 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxcjs\" (UniqueName: \"kubernetes.io/projected/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-kube-api-access-zxcjs\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882496 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882406 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-wtmp\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882496 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882449 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxcjs\" (UniqueName: \"kubernetes.io/projected/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-kube-api-access-zxcjs\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882702 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882573 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-root\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882702 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882603 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-textfile\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882702 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882630 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882702 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882625 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-wtmp\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882702 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882676 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-sys\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882703 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-root\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882717 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-tls\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882761 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882778 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-sys\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.882946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882808 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-metrics-client-ca\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.883219 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.882998 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-textfile\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.883410 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.883391 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-metrics-client-ca\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.883508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.883484 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.885213 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.885193 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.885349 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.885322 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-node-exporter-tls\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.942959 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.942916 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxcjs\" (UniqueName: \"kubernetes.io/projected/4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4-kube-api-access-zxcjs\") pod \"node-exporter-wbwhr\" (UID: \"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4\") " pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:55.946675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:55.946647 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wbwhr" Oct 31 03:28:56.286102 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:56.286056 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wbwhr" event={"ID":"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4","Type":"ContainerStarted","Data":"5f36230a762e1d67698b6591044ecaca61147ccec8aea64be934145f5aa3e176"} Oct 31 03:28:58.202364 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:58.202325 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bh9xh" Oct 31 03:28:58.295252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:58.295181 4426 generic.go:356] "Generic (PLEG): container finished" podID="4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4" containerID="0177da3d6a4254e1bee55845949384c845ba9dc875ffbf86d7686f740ff6c4e7" exitCode=0 Oct 31 03:28:58.295252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:58.295248 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wbwhr" event={"ID":"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4","Type":"ContainerDied","Data":"0177da3d6a4254e1bee55845949384c845ba9dc875ffbf86d7686f740ff6c4e7"} Oct 31 03:28:59.301090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:59.301046 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wbwhr" event={"ID":"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4","Type":"ContainerStarted","Data":"1799fae895904b8771896f67af956f1624ffc9e4ff5feee33d614544be94ad7d"} Oct 31 03:28:59.301598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:59.301113 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wbwhr" event={"ID":"4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4","Type":"ContainerStarted","Data":"6108e400f7ae60266b0aafaafb1c4a4b255a6d90ee18f0049974e9df3a422483"} Oct 31 03:28:59.459730 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:28:59.459613 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wbwhr" podStartSLOduration=2.311331269 podStartE2EDuration="4.459596532s" totalImagesPullingTime="2.05902445s" totalInitContainerRuntime="89.240813ms" isStatefulPod=true podCreationTimestamp="2026-10-31 03:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:59.459074246 +0000 UTC m=+36.459699492" watchObservedRunningTime="2026-10-31 03:28:59.459596532 +0000 UTC m=+36.460221750" Oct 31 03:29:00.250426 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.250386 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl"] Oct 31 03:29:00.253475 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.253445 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" Oct 31 03:29:00.321807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.321776 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tz9d6\"" Oct 31 03:29:00.368157 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.368104 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 03:29:00.368860 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.368832 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl"] Oct 31 03:29:00.417499 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.417458 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppj6s\" (UniqueName: \"kubernetes.io/projected/a3beef8c-8001-47b8-a915-846eecb48f2d-kube-api-access-ppj6s\") pod \"monitoring-plugin-77f4f76594-q8wfl\" (UID: \"a3beef8c-8001-47b8-a915-846eecb48f2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" Oct 31 03:29:00.417766 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.417728 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a3beef8c-8001-47b8-a915-846eecb48f2d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-q8wfl\" (UID: \"a3beef8c-8001-47b8-a915-846eecb48f2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" Oct 31 03:29:00.518287 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.518206 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppj6s\" (UniqueName: \"kubernetes.io/projected/a3beef8c-8001-47b8-a915-846eecb48f2d-kube-api-access-ppj6s\") pod \"monitoring-plugin-77f4f76594-q8wfl\" (UID: \"a3beef8c-8001-47b8-a915-846eecb48f2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" Oct 31 03:29:00.518287 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.518275 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a3beef8c-8001-47b8-a915-846eecb48f2d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-q8wfl\" (UID: \"a3beef8c-8001-47b8-a915-846eecb48f2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" Oct 31 03:29:00.521082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.521052 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a3beef8c-8001-47b8-a915-846eecb48f2d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-q8wfl\" (UID: \"a3beef8c-8001-47b8-a915-846eecb48f2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" Oct 31 03:29:00.576065 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.576034 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppj6s\" (UniqueName: \"kubernetes.io/projected/a3beef8c-8001-47b8-a915-846eecb48f2d-kube-api-access-ppj6s\") pod \"monitoring-plugin-77f4f76594-q8wfl\" (UID: \"a3beef8c-8001-47b8-a915-846eecb48f2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" Oct 31 03:29:00.866645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:00.866564 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" Oct 31 03:29:01.239032 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:01.238989 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl"] Oct 31 03:29:01.243107 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:29:01.243072 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3beef8c_8001_47b8_a915_846eecb48f2d.slice/crio-4d6d1a41ffe69c2a2afef69880be59bf1bb3fa96ce87de17707606cc86a775e0 WatchSource:0}: Error finding container 4d6d1a41ffe69c2a2afef69880be59bf1bb3fa96ce87de17707606cc86a775e0: Status 404 returned error can't find the container with id 4d6d1a41ffe69c2a2afef69880be59bf1bb3fa96ce87de17707606cc86a775e0 Oct 31 03:29:01.310382 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:01.310348 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" event={"ID":"a3beef8c-8001-47b8-a915-846eecb48f2d","Type":"ContainerStarted","Data":"4d6d1a41ffe69c2a2afef69880be59bf1bb3fa96ce87de17707606cc86a775e0"} Oct 31 03:29:02.316774 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:02.316736 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j9lnw" event={"ID":"c7216fab-cc51-42e9-990a-3690b21846f5","Type":"ContainerStarted","Data":"92e6f5fd8432ef5374877837b6f05ec6a00ac175cad107858e7c4175a3f3dae7"} Oct 31 03:29:02.317719 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:02.317696 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-j9lnw" Oct 31 03:29:02.334219 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:02.334188 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-j9lnw" Oct 31 03:29:02.519163 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:02.519080 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-j9lnw" podStartSLOduration=2.528086224 podStartE2EDuration="19.519059782s" totalImagesPullingTime="16.990973558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:02.472558083 +0000 UTC m=+39.473183317" watchObservedRunningTime="2026-10-31 03:29:02.519059782 +0000 UTC m=+39.519685010" Oct 31 03:29:02.855480 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:29:02.855443 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:29:04.327478 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:04.327424 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" event={"ID":"a3beef8c-8001-47b8-a915-846eecb48f2d","Type":"ContainerStarted","Data":"a895b5910cb66751ceeb140806f7f81824992cd17f0c23b672d5f0c2273b9b19"} Oct 31 03:29:04.327967 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:04.327718 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" Oct 31 03:29:04.333289 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:04.333263 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" Oct 31 03:29:04.469934 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:04.469831 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q8wfl" podStartSLOduration=1.563596771 podStartE2EDuration="4.469811514s" totalImagesPullingTime="2.906214743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:04.469001515 +0000 UTC m=+41.469626743" watchObservedRunningTime="2026-10-31 03:29:04.469811514 +0000 UTC m=+41.470436744" Oct 31 03:29:06.253252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:06.253170 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-676475bf7d-q7pvn" Oct 31 03:29:32.849322 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:29:32.849275 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:29:46.229810 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.229773 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hnt22-469x9"] Oct 31 03:29:46.230428 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.230040 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-hnt22-469x9" podUID="2d2a2662-e327-4427-a3e8-482de3a8ed26" containerName="insights-gathering" containerID="cri-o://7861833dddccd59bc60270491bde6231533a30990a5d10874928513dd24f7490" gracePeriod=30 Oct 31 03:29:46.463509 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.463474 4426 generic.go:356] "Generic (PLEG): container finished" podID="2d2a2662-e327-4427-a3e8-482de3a8ed26" containerID="7861833dddccd59bc60270491bde6231533a30990a5d10874928513dd24f7490" exitCode=2 Oct 31 03:29:46.463643 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.463542 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hnt22-469x9" event={"ID":"2d2a2662-e327-4427-a3e8-482de3a8ed26","Type":"ContainerDied","Data":"7861833dddccd59bc60270491bde6231533a30990a5d10874928513dd24f7490"} Oct 31 03:29:46.463643 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.463594 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hnt22-469x9" event={"ID":"2d2a2662-e327-4427-a3e8-482de3a8ed26","Type":"ContainerDied","Data":"093dbcd91688b4f46084730b8bf5451b0e6af412eaf61f864416411886017e0e"} Oct 31 03:29:46.463643 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.463611 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="093dbcd91688b4f46084730b8bf5451b0e6af412eaf61f864416411886017e0e" Oct 31 03:29:46.466676 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.466658 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:29:46.538857 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.538776 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2a2662-e327-4427-a3e8-482de3a8ed26-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2a2662-e327-4427-a3e8-482de3a8ed26-archives-path\") pod \"2d2a2662-e327-4427-a3e8-482de3a8ed26\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " Oct 31 03:29:46.538857 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.538815 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2a2662-e327-4427-a3e8-482de3a8ed26-kube-api-access-4dkdj\" (UniqueName: \"kubernetes.io/projected/2d2a2662-e327-4427-a3e8-482de3a8ed26-kube-api-access-4dkdj\") pod \"2d2a2662-e327-4427-a3e8-482de3a8ed26\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " Oct 31 03:29:46.538857 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.538848 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-service-ca-bundle\") pod \"2d2a2662-e327-4427-a3e8-482de3a8ed26\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " Oct 31 03:29:46.539062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.538968 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-trusted-ca-bundle\") pod \"2d2a2662-e327-4427-a3e8-482de3a8ed26\" (UID: \"2d2a2662-e327-4427-a3e8-482de3a8ed26\") " Oct 31 03:29:46.539241 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.539177 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2a2662-e327-4427-a3e8-482de3a8ed26-archives-path" pod "2d2a2662-e327-4427-a3e8-482de3a8ed26" (UID: "2d2a2662-e327-4427-a3e8-482de3a8ed26"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:29:46.539241 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.539197 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-service-ca-bundle" pod "2d2a2662-e327-4427-a3e8-482de3a8ed26" (UID: "2d2a2662-e327-4427-a3e8-482de3a8ed26"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:29:46.539368 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.539339 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-trusted-ca-bundle" pod "2d2a2662-e327-4427-a3e8-482de3a8ed26" (UID: "2d2a2662-e327-4427-a3e8-482de3a8ed26"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:29:46.541051 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.541032 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2a2662-e327-4427-a3e8-482de3a8ed26-kube-api-access-4dkdj" pod "2d2a2662-e327-4427-a3e8-482de3a8ed26" (UID: "2d2a2662-e327-4427-a3e8-482de3a8ed26"). InnerVolumeSpecName "kube-api-access-4dkdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:29:46.640345 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.640306 4426 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2a2662-e327-4427-a3e8-482de3a8ed26-archives-path\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:46.640345 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.640337 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dkdj\" (UniqueName: \"kubernetes.io/projected/2d2a2662-e327-4427-a3e8-482de3a8ed26-kube-api-access-4dkdj\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:46.640345 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.640353 4426 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-service-ca-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:46.640559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:46.640364 4426 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2a2662-e327-4427-a3e8-482de3a8ed26-trusted-ca-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:47.467010 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:47.466982 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hnt22-469x9" Oct 31 03:29:47.618962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:47.618928 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hnt22-469x9"] Oct 31 03:29:47.664917 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:47.664879 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-hnt22-469x9"] Oct 31 03:29:49.125858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:29:49.125821 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2a2662-e327-4427-a3e8-482de3a8ed26" path="/var/lib/kubelet/pods/2d2a2662-e327-4427-a3e8-482de3a8ed26/volumes" Oct 31 03:30:02.845294 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:30:02.845247 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:30:24.919002 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.918968 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn"] Oct 31 03:30:24.919551 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.919270 4426 cpu_manager.go:414] "containerMap: removing container" podUID="2d2a2662-e327-4427-a3e8-482de3a8ed26" containerName="insights-gathering" Oct 31 03:30:24.919551 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.919281 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2a2662-e327-4427-a3e8-482de3a8ed26" containerName="insights-gathering" Oct 31 03:30:24.919551 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.919329 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2a2662-e327-4427-a3e8-482de3a8ed26" containerName="insights-gathering" Oct 31 03:30:24.922250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.922233 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:24.971318 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.971283 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 03:30:24.995471 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.995437 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db970dc-067c-40d3-b333-031fd723eefe-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:24.995471 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.995471 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-telemeter-client-tls\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:24.995689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.995491 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db970dc-067c-40d3-b333-031fd723eefe-serving-certs-ca-bundle\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:24.995689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.995515 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2bpx\" (UniqueName: \"kubernetes.io/projected/2db970dc-067c-40d3-b333-031fd723eefe-kube-api-access-f2bpx\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:24.995689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.995564 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:24.995689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.995601 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2db970dc-067c-40d3-b333-031fd723eefe-metrics-client-ca\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:24.995894 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.995735 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-federate-client-tls\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:24.995894 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:24.995773 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-secret-telemeter-client\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.064140 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.064085 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 03:30:25.064330 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.064265 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 03:30:25.064330 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.064288 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 03:30:25.064330 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.064298 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 03:30:25.064463 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.064353 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 03:30:25.064507 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.064483 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-x6f25\"" Oct 31 03:30:25.065240 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.065221 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn"] Oct 31 03:30:25.096968 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.096928 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-federate-client-tls\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.096968 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.096969 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-secret-telemeter-client\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.097226 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.097016 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db970dc-067c-40d3-b333-031fd723eefe-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.097226 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.097032 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-telemeter-client-tls\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.097226 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.097050 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db970dc-067c-40d3-b333-031fd723eefe-serving-certs-ca-bundle\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.097226 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.097072 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bpx\" (UniqueName: \"kubernetes.io/projected/2db970dc-067c-40d3-b333-031fd723eefe-kube-api-access-f2bpx\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.097226 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.097099 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.097226 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.097148 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2db970dc-067c-40d3-b333-031fd723eefe-metrics-client-ca\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.098049 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.097971 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2db970dc-067c-40d3-b333-031fd723eefe-metrics-client-ca\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.098049 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.097971 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db970dc-067c-40d3-b333-031fd723eefe-serving-certs-ca-bundle\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.098249 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.098184 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db970dc-067c-40d3-b333-031fd723eefe-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.099640 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.099613 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-secret-telemeter-client\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.099959 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.099937 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-federate-client-tls\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.100053 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.099939 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-telemeter-client-tls\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.100053 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.100007 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2db970dc-067c-40d3-b333-031fd723eefe-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.154396 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.154363 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2bpx\" (UniqueName: \"kubernetes.io/projected/2db970dc-067c-40d3-b333-031fd723eefe-kube-api-access-f2bpx\") pod \"telemeter-client-6bcf86fdd-bl8wn\" (UID: \"2db970dc-067c-40d3-b333-031fd723eefe\") " pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.236928 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.236814 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" Oct 31 03:30:25.653327 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:25.653295 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn"] Oct 31 03:30:25.656596 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:30:25.656571 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db970dc_067c_40d3_b333_031fd723eefe.slice/crio-e2fce195d6cb458c86d41b1c34061286f20e8a8bfca10e86386d34c2f5dc6ea0 WatchSource:0}: Error finding container e2fce195d6cb458c86d41b1c34061286f20e8a8bfca10e86386d34c2f5dc6ea0: Status 404 returned error can't find the container with id e2fce195d6cb458c86d41b1c34061286f20e8a8bfca10e86386d34c2f5dc6ea0 Oct 31 03:30:26.594441 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:26.594401 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" event={"ID":"2db970dc-067c-40d3-b333-031fd723eefe","Type":"ContainerStarted","Data":"e2fce195d6cb458c86d41b1c34061286f20e8a8bfca10e86386d34c2f5dc6ea0"} Oct 31 03:30:28.602344 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:28.602311 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" event={"ID":"2db970dc-067c-40d3-b333-031fd723eefe","Type":"ContainerStarted","Data":"ca686633749b63a9dcd24ce522c532f8f506bac18462269921f5e0914a0478ef"} Oct 31 03:30:29.607986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:29.607888 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" event={"ID":"2db970dc-067c-40d3-b333-031fd723eefe","Type":"ContainerStarted","Data":"aca42c6aeee3bd85863b7b9914c9f534e84ef17210251223c1d9a241ebc15522"} Oct 31 03:30:29.608412 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:29.608001 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" event={"ID":"2db970dc-067c-40d3-b333-031fd723eefe","Type":"ContainerStarted","Data":"39ef023feb6fb81b5a649cbc6366a605e83c2a1bf2a71bc57fe44f20c305bd96"} Oct 31 03:30:29.814730 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:30:29.814668 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6bcf86fdd-bl8wn" podStartSLOduration=2.1772118 podStartE2EDuration="5.814654799s" totalImagesPullingTime="3.637442999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:29.768375 +0000 UTC m=+126.769000228" watchObservedRunningTime="2026-10-31 03:30:29.814654799 +0000 UTC m=+126.815280062" Oct 31 03:30:32.844254 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:30:32.844216 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:31:02.844284 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:31:02.844243 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:31:32.844659 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:31:32.844619 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:32:02.844106 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:32:02.844065 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:32:32.844069 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:32:32.844028 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:33:02.844808 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:33:02.844767 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:33:32.845427 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:33:32.845387 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:34:02.844080 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:34:02.844037 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:34:18.429317 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.429278 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27"] Oct 31 03:34:18.432730 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.432712 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:34:18.528264 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.528230 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 03:34:18.528264 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.528230 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 03:34:18.528505 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.528349 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-d2bz6\"" Oct 31 03:34:18.529328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.529308 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27"] Oct 31 03:34:18.538794 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.538770 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:34:18.538895 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.538804 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg8ph\" (UniqueName: \"kubernetes.io/projected/7e61d857-3dc0-4120-89d1-a9e531947e51-kube-api-access-fg8ph\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:34:18.538895 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.538829 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:34:18.640353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.640319 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:34:18.640353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.640360 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg8ph\" (UniqueName: \"kubernetes.io/projected/7e61d857-3dc0-4120-89d1-a9e531947e51-kube-api-access-fg8ph\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:34:18.640588 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.640381 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:34:18.640706 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.640691 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:34:18.640771 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.640751 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:34:18.698801 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.698724 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg8ph\" (UniqueName: \"kubernetes.io/projected/7e61d857-3dc0-4120-89d1-a9e531947e51-kube-api-access-fg8ph\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:34:18.747691 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:18.747656 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:34:19.113424 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:19.113338 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27"] Oct 31 03:34:19.116838 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:34:19.116808 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e61d857_3dc0_4120_89d1_a9e531947e51.slice/crio-d323b7f118d3b66acc258473dacfcf9031286b0b1a7ddb6becc98cf87896c94e WatchSource:0}: Error finding container d323b7f118d3b66acc258473dacfcf9031286b0b1a7ddb6becc98cf87896c94e: Status 404 returned error can't find the container with id d323b7f118d3b66acc258473dacfcf9031286b0b1a7ddb6becc98cf87896c94e Oct 31 03:34:19.118934 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:19.118913 4426 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:34:19.347860 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:19.347823 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" event={"ID":"7e61d857-3dc0-4120-89d1-a9e531947e51","Type":"ContainerStarted","Data":"d323b7f118d3b66acc258473dacfcf9031286b0b1a7ddb6becc98cf87896c94e"} Oct 31 03:34:20.049203 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.049167 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc"] Oct 31 03:34:20.053762 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.053738 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:34:20.149111 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.149084 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ww4r2\"" Oct 31 03:34:20.149932 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.149910 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc"] Oct 31 03:34:20.151831 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.151806 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:34:20.151960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.151846 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6k7\" (UniqueName: \"kubernetes.io/projected/ccff26f6-b677-45ee-920a-1692c44f4fe9-kube-api-access-kr6k7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:34:20.151960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.151877 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:34:20.152635 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.152616 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t"] Oct 31 03:34:20.156062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.155917 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:34:20.205235 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.205203 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-pbg57\"" Oct 31 03:34:20.215941 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.215902 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t"] Oct 31 03:34:20.252416 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.252374 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:34:20.252607 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.252441 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7kb\" (UniqueName: \"kubernetes.io/projected/c63067d6-40c5-4473-aa7f-dfcb1b52d141-kube-api-access-gb7kb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:34:20.252607 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.252467 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:34:20.252607 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.252531 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:34:20.252781 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.252650 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6k7\" (UniqueName: \"kubernetes.io/projected/ccff26f6-b677-45ee-920a-1692c44f4fe9-kube-api-access-kr6k7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:34:20.252781 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.252732 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:34:20.253238 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.253216 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:34:20.253470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.253446 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:34:20.253726 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.253709 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf"] Oct 31 03:34:20.257079 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.257060 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:34:20.307488 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.307425 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-9hcml\"" Oct 31 03:34:20.311564 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.311532 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6k7\" (UniqueName: \"kubernetes.io/projected/ccff26f6-b677-45ee-920a-1692c44f4fe9-kube-api-access-kr6k7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:34:20.318908 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.318879 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf"] Oct 31 03:34:20.353809 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.353773 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:34:20.353973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.353848 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:34:20.353973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.353930 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrhmc\" (UniqueName: \"kubernetes.io/projected/919ff02d-ecd8-4a04-92fb-92f539b4c676-kube-api-access-nrhmc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:34:20.354093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.353972 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7kb\" (UniqueName: \"kubernetes.io/projected/c63067d6-40c5-4473-aa7f-dfcb1b52d141-kube-api-access-gb7kb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:34:20.354093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.353995 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:34:20.354093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.354065 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:34:20.354288 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.354251 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:34:20.354344 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.354305 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:34:20.368818 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.368793 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8"] Oct 31 03:34:20.368924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.368885 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:34:20.372110 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.372094 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:34:20.412356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.412328 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7kb\" (UniqueName: \"kubernetes.io/projected/c63067d6-40c5-4473-aa7f-dfcb1b52d141-kube-api-access-gb7kb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:34:20.421567 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.421534 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-5js5w\"" Oct 31 03:34:20.432636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.432561 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8"] Oct 31 03:34:20.454749 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.454713 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhmc\" (UniqueName: \"kubernetes.io/projected/919ff02d-ecd8-4a04-92fb-92f539b4c676-kube-api-access-nrhmc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:34:20.454917 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.454769 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:34:20.454917 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.454811 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhf5w\" (UniqueName: \"kubernetes.io/projected/d9690ccb-5acc-47a7-a664-1d655523e64e-kube-api-access-lhf5w\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:34:20.454917 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.454849 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:34:20.454917 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.454883 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:34:20.455158 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.454935 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:34:20.456029 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.455737 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:34:20.456029 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.455989 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:34:20.471926 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.471899 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:34:20.513571 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.513499 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrhmc\" (UniqueName: \"kubernetes.io/projected/919ff02d-ecd8-4a04-92fb-92f539b4c676-kube-api-access-nrhmc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:34:20.555608 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.555580 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:34:20.555753 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.555622 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhf5w\" (UniqueName: \"kubernetes.io/projected/d9690ccb-5acc-47a7-a664-1d655523e64e-kube-api-access-lhf5w\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:34:20.555753 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.555681 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:34:20.556018 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.555995 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:34:20.556106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.556063 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:34:20.573671 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.573603 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:34:20.614199 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.614163 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhf5w\" (UniqueName: \"kubernetes.io/projected/d9690ccb-5acc-47a7-a664-1d655523e64e-kube-api-access-lhf5w\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:34:20.700697 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.700664 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:34:20.740224 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.740190 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc"] Oct 31 03:34:20.845474 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.845411 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t"] Oct 31 03:34:20.848848 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:34:20.848820 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63067d6_40c5_4473_aa7f_dfcb1b52d141.slice/crio-9317b0e911bed12b5e59fff8ee0d78f91aaec745f863d77db459f3be4942a9c8 WatchSource:0}: Error finding container 9317b0e911bed12b5e59fff8ee0d78f91aaec745f863d77db459f3be4942a9c8: Status 404 returned error can't find the container with id 9317b0e911bed12b5e59fff8ee0d78f91aaec745f863d77db459f3be4942a9c8 Oct 31 03:34:20.947629 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:20.947594 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf"] Oct 31 03:34:20.950799 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:34:20.950769 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919ff02d_ecd8_4a04_92fb_92f539b4c676.slice/crio-94f27305b9b682447c8d0d2be175b3294bfff92d9aebbc2b8562af0bfe90ab36 WatchSource:0}: Error finding container 94f27305b9b682447c8d0d2be175b3294bfff92d9aebbc2b8562af0bfe90ab36: Status 404 returned error can't find the container with id 94f27305b9b682447c8d0d2be175b3294bfff92d9aebbc2b8562af0bfe90ab36 Oct 31 03:34:21.072306 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.072273 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8"] Oct 31 03:34:21.076393 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:34:21.076362 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9690ccb_5acc_47a7_a664_1d655523e64e.slice/crio-fe8473080b6af26ebfe6cd91b52fbf2ae69e1625a568f262bbb83d840996cf0a WatchSource:0}: Error finding container fe8473080b6af26ebfe6cd91b52fbf2ae69e1625a568f262bbb83d840996cf0a: Status 404 returned error can't find the container with id fe8473080b6af26ebfe6cd91b52fbf2ae69e1625a568f262bbb83d840996cf0a Oct 31 03:34:21.357261 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.357226 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" event={"ID":"919ff02d-ecd8-4a04-92fb-92f539b4c676","Type":"ContainerStarted","Data":"94f27305b9b682447c8d0d2be175b3294bfff92d9aebbc2b8562af0bfe90ab36"} Oct 31 03:34:21.358542 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.358509 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" event={"ID":"d9690ccb-5acc-47a7-a664-1d655523e64e","Type":"ContainerStarted","Data":"fe8473080b6af26ebfe6cd91b52fbf2ae69e1625a568f262bbb83d840996cf0a"} Oct 31 03:34:21.360011 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.359979 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" event={"ID":"ccff26f6-b677-45ee-920a-1692c44f4fe9","Type":"ContainerStarted","Data":"a081d6fce197db9a1c4d0c919c41d2f9007b952114a786735cd805db28271d1f"} Oct 31 03:34:21.361318 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.361290 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" event={"ID":"c63067d6-40c5-4473-aa7f-dfcb1b52d141","Type":"ContainerStarted","Data":"9317b0e911bed12b5e59fff8ee0d78f91aaec745f863d77db459f3be4942a9c8"} Oct 31 03:34:21.385736 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.385703 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97"] Oct 31 03:34:21.390320 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.390291 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:34:21.439251 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.439216 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-j96pf\"" Oct 31 03:34:21.445312 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.445278 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97"] Oct 31 03:34:21.563867 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.563826 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2tw8\" (UniqueName: \"kubernetes.io/projected/7221fc5a-653b-45cc-bd81-3f87a9816a90-kube-api-access-f2tw8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:34:21.564055 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.563901 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:34:21.564055 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.564019 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:34:21.665095 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.665053 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:34:21.665288 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.665180 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:34:21.665288 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.665248 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2tw8\" (UniqueName: \"kubernetes.io/projected/7221fc5a-653b-45cc-bd81-3f87a9816a90-kube-api-access-f2tw8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:34:21.665618 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.665595 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:34:21.665845 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.665824 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:34:21.723531 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:21.723500 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2tw8\" (UniqueName: \"kubernetes.io/projected/7221fc5a-653b-45cc-bd81-3f87a9816a90-kube-api-access-f2tw8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:34:22.008194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.008148 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:34:22.412701 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.412661 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97"] Oct 31 03:34:22.418763 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:34:22.418731 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7221fc5a_653b_45cc_bd81_3f87a9816a90.slice/crio-e45ec7b2fa1293171438806fe313757599d96f034113920e16338b4001f57476 WatchSource:0}: Error finding container e45ec7b2fa1293171438806fe313757599d96f034113920e16338b4001f57476: Status 404 returned error can't find the container with id e45ec7b2fa1293171438806fe313757599d96f034113920e16338b4001f57476 Oct 31 03:34:22.642116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.642081 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks"] Oct 31 03:34:22.647195 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.646369 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:34:22.696414 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.696281 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-f95cm\"" Oct 31 03:34:22.743637 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.743536 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks"] Oct 31 03:34:22.776901 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.776655 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:34:22.776901 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.776715 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:34:22.776901 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.776782 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7c6k\" (UniqueName: \"kubernetes.io/projected/66b8be12-1874-49a0-acc3-f6a033028a37-kube-api-access-p7c6k\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:34:22.878689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.877624 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:34:22.878689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.877689 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:34:22.878689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.877752 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7c6k\" (UniqueName: \"kubernetes.io/projected/66b8be12-1874-49a0-acc3-f6a033028a37-kube-api-access-p7c6k\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:34:22.878689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.878357 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:34:22.878689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.878594 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:34:22.950353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.949737 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7c6k\" (UniqueName: \"kubernetes.io/projected/66b8be12-1874-49a0-acc3-f6a033028a37-kube-api-access-p7c6k\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:34:22.968987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:22.968951 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:34:23.373421 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:23.373335 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" event={"ID":"7221fc5a-653b-45cc-bd81-3f87a9816a90","Type":"ContainerStarted","Data":"e45ec7b2fa1293171438806fe313757599d96f034113920e16338b4001f57476"} Oct 31 03:34:23.386857 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:23.386821 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks"] Oct 31 03:34:23.433230 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:34:23.425806 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b8be12_1874_49a0_acc3_f6a033028a37.slice/crio-e86cbec6404972fe359b866f7ff2037113ddc5252595aacaa4c294ebef242198 WatchSource:0}: Error finding container e86cbec6404972fe359b866f7ff2037113ddc5252595aacaa4c294ebef242198: Status 404 returned error can't find the container with id e86cbec6404972fe359b866f7ff2037113ddc5252595aacaa4c294ebef242198 Oct 31 03:34:24.382965 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:24.382924 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" event={"ID":"66b8be12-1874-49a0-acc3-f6a033028a37","Type":"ContainerStarted","Data":"e86cbec6404972fe359b866f7ff2037113ddc5252595aacaa4c294ebef242198"} Oct 31 03:34:25.426945 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.426894 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8"] Oct 31 03:34:25.433032 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.432407 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:34:25.493880 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.493842 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8"] Oct 31 03:34:25.603157 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.603096 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:34:25.603353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.603196 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8twj\" (UniqueName: \"kubernetes.io/projected/6c1351f6-f856-42e2-b702-8dbf4b943f5f-kube-api-access-h8twj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:34:25.603353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.603256 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:34:25.705221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.703969 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:34:25.705221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.704046 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:34:25.705221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.704137 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8twj\" (UniqueName: \"kubernetes.io/projected/6c1351f6-f856-42e2-b702-8dbf4b943f5f-kube-api-access-h8twj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:34:25.705221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.704957 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:34:25.705760 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.705712 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:34:25.769635 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:25.769562 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8twj\" (UniqueName: \"kubernetes.io/projected/6c1351f6-f856-42e2-b702-8dbf4b943f5f-kube-api-access-h8twj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:34:26.055340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:26.055251 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:34:26.468595 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:26.468470 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8"] Oct 31 03:34:26.481037 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:34:26.480996 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1351f6_f856_42e2_b702_8dbf4b943f5f.slice/crio-427c2a24a986f36d0659a1b91842ef9713e9ecc017d52a2e0dba7645f219daba WatchSource:0}: Error finding container 427c2a24a986f36d0659a1b91842ef9713e9ecc017d52a2e0dba7645f219daba: Status 404 returned error can't find the container with id 427c2a24a986f36d0659a1b91842ef9713e9ecc017d52a2e0dba7645f219daba Oct 31 03:34:27.412800 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:27.412718 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" event={"ID":"6c1351f6-f856-42e2-b702-8dbf4b943f5f","Type":"ContainerStarted","Data":"427c2a24a986f36d0659a1b91842ef9713e9ecc017d52a2e0dba7645f219daba"} Oct 31 03:34:28.251750 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.249660 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s"] Oct 31 03:34:28.254174 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.254088 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:34:28.278244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.276342 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2"] Oct 31 03:34:28.281231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.280804 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:34:28.324928 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.324895 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb"] Oct 31 03:34:28.330794 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.330614 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s"] Oct 31 03:34:28.331032 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.330726 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:34:28.353716 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.353691 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6"] Oct 31 03:34:28.358000 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.357964 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:34:28.377570 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.377539 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb"] Oct 31 03:34:28.381923 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.381897 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2"] Oct 31 03:34:28.401081 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.401045 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6"] Oct 31 03:34:28.427821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.427467 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:34:28.427821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.427510 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5cm\" (UniqueName: \"kubernetes.io/projected/5b323f7a-fc47-494d-a832-ee7096a5cc85-kube-api-access-wq5cm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:34:28.427821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.427532 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:34:28.427821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.427560 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:34:28.427821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.427582 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvtw\" (UniqueName: \"kubernetes.io/projected/6a8607f3-58c0-42db-8f69-5a17b8d22b93-kube-api-access-wlvtw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:34:28.427821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.427606 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:34:28.427821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.427622 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:34:28.427821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.427635 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:34:28.427821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.427652 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft586\" (UniqueName: \"kubernetes.io/projected/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-kube-api-access-ft586\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.528893 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.528944 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.528972 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.529002 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft586\" (UniqueName: \"kubernetes.io/projected/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-kube-api-access-ft586\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.529039 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8xq\" (UniqueName: \"kubernetes.io/projected/8230184b-11d2-41bc-9725-69c1b9bd4b63-kube-api-access-6x8xq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.529101 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.529150 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.529196 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5cm\" (UniqueName: \"kubernetes.io/projected/5b323f7a-fc47-494d-a832-ee7096a5cc85-kube-api-access-wq5cm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.529228 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.529271 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.529311 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvtw\" (UniqueName: \"kubernetes.io/projected/6a8607f3-58c0-42db-8f69-5a17b8d22b93-kube-api-access-wlvtw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.529349 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.529803 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.530112 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.530403 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.531049 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.531522 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:34:28.532061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.531752 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:34:28.604310 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.604156 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft586\" (UniqueName: \"kubernetes.io/projected/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-kube-api-access-ft586\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:34:28.604866 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.604785 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5cm\" (UniqueName: \"kubernetes.io/projected/5b323f7a-fc47-494d-a832-ee7096a5cc85-kube-api-access-wq5cm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:34:28.606096 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.606062 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvtw\" (UniqueName: \"kubernetes.io/projected/6a8607f3-58c0-42db-8f69-5a17b8d22b93-kube-api-access-wlvtw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:34:28.630133 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.630077 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:34:28.630279 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.630207 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:34:28.630279 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.630260 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8xq\" (UniqueName: \"kubernetes.io/projected/8230184b-11d2-41bc-9725-69c1b9bd4b63-kube-api-access-6x8xq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:34:28.631018 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.630870 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:34:28.631445 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.631419 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:34:28.659040 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.659004 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:34:28.714045 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.713999 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8xq\" (UniqueName: \"kubernetes.io/projected/8230184b-11d2-41bc-9725-69c1b9bd4b63-kube-api-access-6x8xq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:34:28.877811 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.877470 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:34:28.904984 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.904690 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:34:28.987188 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:28.979601 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:34:29.098431 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.096258 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb"] Oct 31 03:34:29.301279 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.301230 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s"] Oct 31 03:34:29.378494 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.378454 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2"] Oct 31 03:34:29.394221 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:34:29.394186 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8607f3_58c0_42db_8f69_5a17b8d22b93.slice/crio-6e8fe4042f4fe10ea452ad03a673a72efb3d3cd488b64af917ac47de1f596d13 WatchSource:0}: Error finding container 6e8fe4042f4fe10ea452ad03a673a72efb3d3cd488b64af917ac47de1f596d13: Status 404 returned error can't find the container with id 6e8fe4042f4fe10ea452ad03a673a72efb3d3cd488b64af917ac47de1f596d13 Oct 31 03:34:29.431649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.431611 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6"] Oct 31 03:34:29.439084 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.439046 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" event={"ID":"6a8607f3-58c0-42db-8f69-5a17b8d22b93","Type":"ContainerStarted","Data":"6e8fe4042f4fe10ea452ad03a673a72efb3d3cd488b64af917ac47de1f596d13"} Oct 31 03:34:29.459236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.458000 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" event={"ID":"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636","Type":"ContainerStarted","Data":"4c24e7cb85bfceba1da0f9c62db869d4c9b41b6eec4f9ee5670759ee3508945f"} Oct 31 03:34:29.472578 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.472537 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" event={"ID":"5b323f7a-fc47-494d-a832-ee7096a5cc85","Type":"ContainerStarted","Data":"ecf518f4f09a23471fadaf5f756d1cc9c69a9fb69badfa8b77c193fac0eac6a8"} Oct 31 03:34:29.496956 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.495518 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh"] Oct 31 03:34:29.503686 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.502776 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.552376 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.552342 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 03:34:29.553184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.553116 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wk9x4\"" Oct 31 03:34:29.556195 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.555903 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 03:34:29.566565 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.566322 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh"] Oct 31 03:34:29.652522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.652484 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.652695 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.652550 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.652695 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.652580 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.652695 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.652601 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf6t4\" (UniqueName: \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-kube-api-access-sf6t4\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.652695 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.652624 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.652695 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.652668 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.652695 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.652692 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.652946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.652713 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.652946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.652748 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.659099 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.659061 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 03:34:29.754327 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.754280 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.754583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.754336 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf6t4\" (UniqueName: \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-kube-api-access-sf6t4\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.754583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.754368 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.754583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.754420 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.754583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.754451 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.754583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.754484 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.754583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.754534 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.754583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.754576 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.754961 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.754615 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.755164 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.755115 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.757191 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.755409 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.757191 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.756391 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.757191 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.757011 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.757191 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.757085 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.765953 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.765879 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.771555 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.771479 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.811072 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.809149 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht"] Oct 31 03:34:29.816669 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.815514 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:29.830601 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.830507 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf6t4\" (UniqueName: \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-kube-api-access-sf6t4\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.832775 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.832681 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-kjrmh\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:29.872962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.872912 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-wwhzv\"" Oct 31 03:34:29.873770 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.873689 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht"] Oct 31 03:34:29.957398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.956718 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:29.957398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.956787 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:29.957398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.956905 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f118b0-a36e-4372-8162-8e9e19d7441b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:29.957398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.956960 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:29.957398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.956985 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:29.957398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.957010 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:29.957398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.957036 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dgfh\" (UniqueName: \"kubernetes.io/projected/f5f118b0-a36e-4372-8162-8e9e19d7441b-kube-api-access-5dgfh\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:29.957398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.957082 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:29.957398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:29.957147 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.057962 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.058026 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.058073 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f118b0-a36e-4372-8162-8e9e19d7441b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.058136 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.058163 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.058193 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.058217 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dgfh\" (UniqueName: \"kubernetes.io/projected/f5f118b0-a36e-4372-8162-8e9e19d7441b-kube-api-access-5dgfh\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.058266 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.058296 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.059001 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.059076 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.059624 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.059822 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f118b0-a36e-4372-8162-8e9e19d7441b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.060147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.059855 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.063111 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.063087 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.065853 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.065809 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.122907 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.122624 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:30.124832 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.124753 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dgfh\" (UniqueName: \"kubernetes.io/projected/f5f118b0-a36e-4372-8162-8e9e19d7441b-kube-api-access-5dgfh\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.128566 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.128514 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f118b0-a36e-4372-8162-8e9e19d7441b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht\" (UID: \"f5f118b0-a36e-4372-8162-8e9e19d7441b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.137045 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.136802 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:30.481551 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.481420 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" event={"ID":"8230184b-11d2-41bc-9725-69c1b9bd4b63","Type":"ContainerStarted","Data":"7cbb4c5b8efcdaaba0c2958749aca3733aa42616c75d404801ab80ec13ec356c"} Oct 31 03:34:30.555498 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.555390 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh"] Oct 31 03:34:30.605408 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:34:30.604211 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41b032c_a9c2_4aac_b4d9_9d87560ba1ea.slice/crio-d8e57b70191b4aa2d122c2de977349850f3286e7db93b68aa3f73766fb5093e0 WatchSource:0}: Error finding container d8e57b70191b4aa2d122c2de977349850f3286e7db93b68aa3f73766fb5093e0: Status 404 returned error can't find the container with id d8e57b70191b4aa2d122c2de977349850f3286e7db93b68aa3f73766fb5093e0 Oct 31 03:34:30.605408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:30.605317 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht"] Oct 31 03:34:30.619737 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:34:30.619648 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f118b0_a36e_4372_8162_8e9e19d7441b.slice/crio-57a46d74233ffa16fdcb5e8bb42aecc0baf06ce04da821df36c3426f0c30e6dc WatchSource:0}: Error finding container 57a46d74233ffa16fdcb5e8bb42aecc0baf06ce04da821df36c3426f0c30e6dc: Status 404 returned error can't find the container with id 57a46d74233ffa16fdcb5e8bb42aecc0baf06ce04da821df36c3426f0c30e6dc Oct 31 03:34:31.495829 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:31.495790 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" event={"ID":"f5f118b0-a36e-4372-8162-8e9e19d7441b","Type":"ContainerStarted","Data":"57a46d74233ffa16fdcb5e8bb42aecc0baf06ce04da821df36c3426f0c30e6dc"} Oct 31 03:34:31.503013 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:31.502976 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" event={"ID":"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea","Type":"ContainerStarted","Data":"d8e57b70191b4aa2d122c2de977349850f3286e7db93b68aa3f73766fb5093e0"} Oct 31 03:34:32.874520 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:34:32.874475 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:34:35.566086 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:35.565891 4426 generic.go:356] "Generic (PLEG): container finished" podID="7e61d857-3dc0-4120-89d1-a9e531947e51" containerID="bf2c34c952572ba98bcb69aa43d0797d8e31b3f03b3812c2c9fceacc5c3667a3" exitCode=0 Oct 31 03:34:35.566086 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:35.566004 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" event={"ID":"7e61d857-3dc0-4120-89d1-a9e531947e51","Type":"ContainerDied","Data":"bf2c34c952572ba98bcb69aa43d0797d8e31b3f03b3812c2c9fceacc5c3667a3"} Oct 31 03:34:36.583596 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:36.583553 4426 generic.go:356] "Generic (PLEG): container finished" podID="66b8be12-1874-49a0-acc3-f6a033028a37" containerID="2f7a23add1792915c5f2a861f0eecf35f010c1ae3995adafc64edda5e270e701" exitCode=0 Oct 31 03:34:36.584161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:36.583679 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" event={"ID":"66b8be12-1874-49a0-acc3-f6a033028a37","Type":"ContainerDied","Data":"2f7a23add1792915c5f2a861f0eecf35f010c1ae3995adafc64edda5e270e701"} Oct 31 03:34:41.647985 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:41.647951 4426 generic.go:356] "Generic (PLEG): container finished" podID="6c1351f6-f856-42e2-b702-8dbf4b943f5f" containerID="052a0d985020a1c23e813168ce0dd2423da4272055b82510a1da2cfa44a63a5e" exitCode=0 Oct 31 03:34:41.648502 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:41.648013 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" event={"ID":"6c1351f6-f856-42e2-b702-8dbf4b943f5f","Type":"ContainerDied","Data":"052a0d985020a1c23e813168ce0dd2423da4272055b82510a1da2cfa44a63a5e"} Oct 31 03:34:42.681020 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:42.679507 4426 generic.go:356] "Generic (PLEG): container finished" podID="919ff02d-ecd8-4a04-92fb-92f539b4c676" containerID="39f40d545ad8306096839ed489b157601604b8178e70187d6c744794ab19e11f" exitCode=0 Oct 31 03:34:42.681020 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:42.680740 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" event={"ID":"919ff02d-ecd8-4a04-92fb-92f539b4c676","Type":"ContainerDied","Data":"39f40d545ad8306096839ed489b157601604b8178e70187d6c744794ab19e11f"} Oct 31 03:34:43.680204 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.680166 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:34:43.680321 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.680255 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:34:43.680321 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.680298 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:34:43.692663 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.691748 4426 generic.go:356] "Generic (PLEG): container finished" podID="ccff26f6-b677-45ee-920a-1692c44f4fe9" containerID="835d660d6072a2df036c88d2994a24e235a673575ca71c53947b6559bbb44b0e" exitCode=0 Oct 31 03:34:43.692663 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.691864 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" event={"ID":"ccff26f6-b677-45ee-920a-1692c44f4fe9","Type":"ContainerDied","Data":"835d660d6072a2df036c88d2994a24e235a673575ca71c53947b6559bbb44b0e"} Oct 31 03:34:43.701052 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.700354 4426 generic.go:356] "Generic (PLEG): container finished" podID="c63067d6-40c5-4473-aa7f-dfcb1b52d141" containerID="e915db2f7976e0dffbda77832ccc433137978ed0caae72a40c10f250b99727af" exitCode=0 Oct 31 03:34:43.701052 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.700450 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" event={"ID":"c63067d6-40c5-4473-aa7f-dfcb1b52d141","Type":"ContainerDied","Data":"e915db2f7976e0dffbda77832ccc433137978ed0caae72a40c10f250b99727af"} Oct 31 03:34:43.703635 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.703606 4426 generic.go:356] "Generic (PLEG): container finished" podID="5b323f7a-fc47-494d-a832-ee7096a5cc85" containerID="168ab60af33b4a66f035129df152d970297e5cad7515d06d3100edd435cc3eeb" exitCode=0 Oct 31 03:34:43.703768 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.703743 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" event={"ID":"5b323f7a-fc47-494d-a832-ee7096a5cc85","Type":"ContainerDied","Data":"168ab60af33b4a66f035129df152d970297e5cad7515d06d3100edd435cc3eeb"} Oct 31 03:34:43.707160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.707118 4426 generic.go:356] "Generic (PLEG): container finished" podID="d9690ccb-5acc-47a7-a664-1d655523e64e" containerID="b6918af646a22dd2b32fd195989ff39bc938c04a6bf8ada73b34bd1ec90f1d98" exitCode=0 Oct 31 03:34:43.707258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.707179 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" event={"ID":"d9690ccb-5acc-47a7-a664-1d655523e64e","Type":"ContainerDied","Data":"b6918af646a22dd2b32fd195989ff39bc938c04a6bf8ada73b34bd1ec90f1d98"} Oct 31 03:34:43.791885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.791833 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:34:43.792019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.791930 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:34:43.792019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:43.791974 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:34:44.713526 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.713492 4426 generic.go:356] "Generic (PLEG): container finished" podID="8230184b-11d2-41bc-9725-69c1b9bd4b63" containerID="29ee183e12e43f88ec8fd2bcfcd1261dda6b3f4b15fd2e0025914873a55548ae" exitCode=0 Oct 31 03:34:44.713943 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.713574 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" event={"ID":"8230184b-11d2-41bc-9725-69c1b9bd4b63","Type":"ContainerDied","Data":"29ee183e12e43f88ec8fd2bcfcd1261dda6b3f4b15fd2e0025914873a55548ae"} Oct 31 03:34:44.715263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.715241 4426 generic.go:356] "Generic (PLEG): container finished" podID="7221fc5a-653b-45cc-bd81-3f87a9816a90" containerID="2531dcc0894035692a3369111354d667a1a7aecc98f976a2b05c0caf95794c9c" exitCode=0 Oct 31 03:34:44.715372 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.715334 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" event={"ID":"7221fc5a-653b-45cc-bd81-3f87a9816a90","Type":"ContainerDied","Data":"2531dcc0894035692a3369111354d667a1a7aecc98f976a2b05c0caf95794c9c"} Oct 31 03:34:44.717703 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.717670 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" event={"ID":"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea","Type":"ContainerStarted","Data":"7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b"} Oct 31 03:34:44.719615 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.719590 4426 generic.go:356] "Generic (PLEG): container finished" podID="7e61d857-3dc0-4120-89d1-a9e531947e51" containerID="d78b549aa48b72063810f4f62232c774465914e2d978e9ef34d4952542594be7" exitCode=0 Oct 31 03:34:44.719721 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.719667 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" event={"ID":"7e61d857-3dc0-4120-89d1-a9e531947e51","Type":"ContainerDied","Data":"d78b549aa48b72063810f4f62232c774465914e2d978e9ef34d4952542594be7"} Oct 31 03:34:44.721338 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.721319 4426 generic.go:356] "Generic (PLEG): container finished" podID="6a8607f3-58c0-42db-8f69-5a17b8d22b93" containerID="89df33a5800350b59e762014f962549b2abd3504d1a65c2c6b04be196326ab3c" exitCode=0 Oct 31 03:34:44.721413 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.721392 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" event={"ID":"6a8607f3-58c0-42db-8f69-5a17b8d22b93","Type":"ContainerDied","Data":"89df33a5800350b59e762014f962549b2abd3504d1a65c2c6b04be196326ab3c"} Oct 31 03:34:44.723839 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.723786 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" event={"ID":"f5f118b0-a36e-4372-8162-8e9e19d7441b","Type":"ContainerStarted","Data":"c131bf71dff192ba75950b68fb65f4f28ce803dbed48a0a941fd668125d9d77f"} Oct 31 03:34:44.725395 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.725371 4426 generic.go:356] "Generic (PLEG): container finished" podID="bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" containerID="58f52f98b5749db1c67fdca3e8e387a0d23a8ba0aff56136f97071557ac12aad" exitCode=0 Oct 31 03:34:44.725502 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.725448 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" event={"ID":"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636","Type":"ContainerDied","Data":"58f52f98b5749db1c67fdca3e8e387a0d23a8ba0aff56136f97071557ac12aad"} Oct 31 03:34:44.727473 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.727451 4426 generic.go:356] "Generic (PLEG): container finished" podID="66b8be12-1874-49a0-acc3-f6a033028a37" containerID="90802a76e412451b0327da558a7ed044581c136cffbe93f7a482e28821fd258c" exitCode=0 Oct 31 03:34:44.727567 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:44.727525 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" event={"ID":"66b8be12-1874-49a0-acc3-f6a033028a37","Type":"ContainerDied","Data":"90802a76e412451b0327da558a7ed044581c136cffbe93f7a482e28821fd258c"} Oct 31 03:34:45.127570 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.127539 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:45.127729 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.127596 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:45.129043 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.129022 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:45.137558 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.137522 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:45.137666 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.137580 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:45.143271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.143246 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:45.156939 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.156889 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" podStartSLOduration=2.986727237 podStartE2EDuration="16.156875117s" totalImagesPullingTime="13.17014788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:45.155075535 +0000 UTC m=+382.155700765" watchObservedRunningTime="2026-10-31 03:34:45.156875117 +0000 UTC m=+382.157500345" Oct 31 03:34:45.604175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.604104 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" podStartSLOduration=3.55135177 podStartE2EDuration="16.604083882s" totalImagesPullingTime="13.052732112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:45.601705186 +0000 UTC m=+382.602330416" watchObservedRunningTime="2026-10-31 03:34:45.604083882 +0000 UTC m=+382.604709111" Oct 31 03:34:45.749367 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.749333 4426 generic.go:356] "Generic (PLEG): container finished" podID="5b323f7a-fc47-494d-a832-ee7096a5cc85" containerID="39cd050a91cb0fec9da13945c4a4557b13503f63a7f69457733165ae63b27a2c" exitCode=0 Oct 31 03:34:45.749746 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.749418 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" event={"ID":"5b323f7a-fc47-494d-a832-ee7096a5cc85","Type":"ContainerDied","Data":"39cd050a91cb0fec9da13945c4a4557b13503f63a7f69457733165ae63b27a2c"} Oct 31 03:34:45.751461 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.751444 4426 generic.go:356] "Generic (PLEG): container finished" podID="919ff02d-ecd8-4a04-92fb-92f539b4c676" containerID="87eb9d1db27e1473117870c94c3205530238ba50d2353167352f03e5062f8c49" exitCode=0 Oct 31 03:34:45.751576 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.751527 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" event={"ID":"919ff02d-ecd8-4a04-92fb-92f539b4c676","Type":"ContainerDied","Data":"87eb9d1db27e1473117870c94c3205530238ba50d2353167352f03e5062f8c49"} Oct 31 03:34:45.753547 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.753526 4426 generic.go:356] "Generic (PLEG): container finished" podID="d9690ccb-5acc-47a7-a664-1d655523e64e" containerID="d175bc30c7c24c43e33dd2bfd7397a3574e18a391dc41f537bc1008e06ebe5ff" exitCode=0 Oct 31 03:34:45.753621 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.753605 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" event={"ID":"d9690ccb-5acc-47a7-a664-1d655523e64e","Type":"ContainerDied","Data":"d175bc30c7c24c43e33dd2bfd7397a3574e18a391dc41f537bc1008e06ebe5ff"} Oct 31 03:34:45.755650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.755630 4426 generic.go:356] "Generic (PLEG): container finished" podID="ccff26f6-b677-45ee-920a-1692c44f4fe9" containerID="fe65cbc8d2e43e827f58a6e51ba968d336c52fd16fdc862248e4746fc6344033" exitCode=0 Oct 31 03:34:45.755744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.755710 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" event={"ID":"ccff26f6-b677-45ee-920a-1692c44f4fe9","Type":"ContainerDied","Data":"fe65cbc8d2e43e827f58a6e51ba968d336c52fd16fdc862248e4746fc6344033"} Oct 31 03:34:45.757700 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.757677 4426 generic.go:356] "Generic (PLEG): container finished" podID="6c1351f6-f856-42e2-b702-8dbf4b943f5f" containerID="58a204290e24e6f11af261c3c465e9d3a801791d5386d72222d08f8c0bbbc4a3" exitCode=0 Oct 31 03:34:45.757795 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.757744 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" event={"ID":"6c1351f6-f856-42e2-b702-8dbf4b943f5f","Type":"ContainerDied","Data":"58a204290e24e6f11af261c3c465e9d3a801791d5386d72222d08f8c0bbbc4a3"} Oct 31 03:34:45.759641 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.759619 4426 generic.go:356] "Generic (PLEG): container finished" podID="c63067d6-40c5-4473-aa7f-dfcb1b52d141" containerID="51117636a34c0625e8d11039d42d4d5e0eb11826ef361401c2b7f4d1c729b2dc" exitCode=0 Oct 31 03:34:45.759719 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.759700 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" event={"ID":"c63067d6-40c5-4473-aa7f-dfcb1b52d141","Type":"ContainerDied","Data":"51117636a34c0625e8d11039d42d4d5e0eb11826ef361401c2b7f4d1c729b2dc"} Oct 31 03:34:45.761388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.761347 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht" Oct 31 03:34:45.761481 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:45.761468 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:34:46.769848 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:46.769667 4426 generic.go:356] "Generic (PLEG): container finished" podID="8230184b-11d2-41bc-9725-69c1b9bd4b63" containerID="77e0026f1e8fcb02765464df31923a50cabcb319b55c151974646c0a143e55ef" exitCode=0 Oct 31 03:34:46.769848 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:46.769771 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" event={"ID":"8230184b-11d2-41bc-9725-69c1b9bd4b63","Type":"ContainerDied","Data":"77e0026f1e8fcb02765464df31923a50cabcb319b55c151974646c0a143e55ef"} Oct 31 03:34:46.775859 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:46.775069 4426 generic.go:356] "Generic (PLEG): container finished" podID="7221fc5a-653b-45cc-bd81-3f87a9816a90" containerID="ae2fcaf08aefa098e1be0e46fd4d3c60bf139ff5be8d5048e14c152194e82214" exitCode=0 Oct 31 03:34:46.775859 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:46.775173 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" event={"ID":"7221fc5a-653b-45cc-bd81-3f87a9816a90","Type":"ContainerDied","Data":"ae2fcaf08aefa098e1be0e46fd4d3c60bf139ff5be8d5048e14c152194e82214"} Oct 31 03:34:46.782182 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:46.781541 4426 generic.go:356] "Generic (PLEG): container finished" podID="6a8607f3-58c0-42db-8f69-5a17b8d22b93" containerID="0110590013fb790782283035ee1c413e1c6d71f46f2ab848c99ee3cda51deddd" exitCode=0 Oct 31 03:34:46.782182 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:46.781630 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" event={"ID":"6a8607f3-58c0-42db-8f69-5a17b8d22b93","Type":"ContainerDied","Data":"0110590013fb790782283035ee1c413e1c6d71f46f2ab848c99ee3cda51deddd"} Oct 31 03:34:46.786869 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:46.785734 4426 generic.go:356] "Generic (PLEG): container finished" podID="bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" containerID="7e134bdb7fd62931b81b9df1493db66c51763d682540235991cfa7fccfb092e0" exitCode=0 Oct 31 03:34:46.786869 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:34:46.786180 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" event={"ID":"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636","Type":"ContainerDied","Data":"7e134bdb7fd62931b81b9df1493db66c51763d682540235991cfa7fccfb092e0"} Oct 31 03:35:02.875215 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:35:02.875159 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:35:08.109946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:08.108988 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" event={"ID":"7e61d857-3dc0-4120-89d1-a9e531947e51","Type":"ContainerStarted","Data":"b5341250deb4b1dc88c09c011d332658faf84616f3607f0e4da075a15f4477c6"} Oct 31 03:35:08.252464 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:08.252405 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" podStartSLOduration=2.681588862 podStartE2EDuration="50.252387249s" totalImagesPullingTime="47.570798387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:08.25136278 +0000 UTC m=+405.251988013" watchObservedRunningTime="2026-10-31 03:35:08.252387249 +0000 UTC m=+405.253012468" Oct 31 03:35:09.118622 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:09.118577 4426 generic.go:356] "Generic (PLEG): container finished" podID="7e61d857-3dc0-4120-89d1-a9e531947e51" containerID="b5341250deb4b1dc88c09c011d332658faf84616f3607f0e4da075a15f4477c6" exitCode=0 Oct 31 03:35:09.119074 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:09.118744 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" event={"ID":"7e61d857-3dc0-4120-89d1-a9e531947e51","Type":"ContainerDied","Data":"b5341250deb4b1dc88c09c011d332658faf84616f3607f0e4da075a15f4477c6"} Oct 31 03:35:09.127030 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:09.126796 4426 generic.go:356] "Generic (PLEG): container finished" podID="66b8be12-1874-49a0-acc3-f6a033028a37" containerID="84de2483d94cba64ccb282a0d242a962d47f67b12f77e4180e04d44b723d7d04" exitCode=0 Oct 31 03:35:09.132556 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:09.132318 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" event={"ID":"66b8be12-1874-49a0-acc3-f6a033028a37","Type":"ContainerDied","Data":"84de2483d94cba64ccb282a0d242a962d47f67b12f77e4180e04d44b723d7d04"} Oct 31 03:35:10.400919 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.400345 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:35:10.422218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.422073 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:35:10.489120 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.489065 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e61d857-3dc0-4120-89d1-a9e531947e51-kube-api-access-fg8ph\" (UniqueName: \"kubernetes.io/projected/7e61d857-3dc0-4120-89d1-a9e531947e51-kube-api-access-fg8ph\") pod \"7e61d857-3dc0-4120-89d1-a9e531947e51\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " Oct 31 03:35:10.489303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.489226 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-bundle\") pod \"7e61d857-3dc0-4120-89d1-a9e531947e51\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " Oct 31 03:35:10.489303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.489290 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-util\" (UniqueName: \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-util\") pod \"7e61d857-3dc0-4120-89d1-a9e531947e51\" (UID: \"7e61d857-3dc0-4120-89d1-a9e531947e51\") " Oct 31 03:35:10.490172 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.489904 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-bundle" pod "7e61d857-3dc0-4120-89d1-a9e531947e51" (UID: "7e61d857-3dc0-4120-89d1-a9e531947e51"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:10.495421 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.495384 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-util" pod "7e61d857-3dc0-4120-89d1-a9e531947e51" (UID: "7e61d857-3dc0-4120-89d1-a9e531947e51"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:10.507770 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.507705 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e61d857-3dc0-4120-89d1-a9e531947e51-kube-api-access-fg8ph" pod "7e61d857-3dc0-4120-89d1-a9e531947e51" (UID: "7e61d857-3dc0-4120-89d1-a9e531947e51"). InnerVolumeSpecName "kube-api-access-fg8ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:10.590760 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.590723 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-util\" (UniqueName: \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-util\") pod \"66b8be12-1874-49a0-acc3-f6a033028a37\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " Oct 31 03:35:10.590957 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.590826 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-bundle\") pod \"66b8be12-1874-49a0-acc3-f6a033028a37\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " Oct 31 03:35:10.590957 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.590910 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b8be12-1874-49a0-acc3-f6a033028a37-kube-api-access-p7c6k\" (UniqueName: \"kubernetes.io/projected/66b8be12-1874-49a0-acc3-f6a033028a37-kube-api-access-p7c6k\") pod \"66b8be12-1874-49a0-acc3-f6a033028a37\" (UID: \"66b8be12-1874-49a0-acc3-f6a033028a37\") " Oct 31 03:35:10.591163 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.591143 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg8ph\" (UniqueName: \"kubernetes.io/projected/7e61d857-3dc0-4120-89d1-a9e531947e51-kube-api-access-fg8ph\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:10.591257 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.591167 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:10.591257 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.591181 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e61d857-3dc0-4120-89d1-a9e531947e51-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:10.593866 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.593795 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-bundle" pod "66b8be12-1874-49a0-acc3-f6a033028a37" (UID: "66b8be12-1874-49a0-acc3-f6a033028a37"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:10.594323 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.594289 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b8be12-1874-49a0-acc3-f6a033028a37-kube-api-access-p7c6k" pod "66b8be12-1874-49a0-acc3-f6a033028a37" (UID: "66b8be12-1874-49a0-acc3-f6a033028a37"). InnerVolumeSpecName "kube-api-access-p7c6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:10.598374 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.598333 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-util" pod "66b8be12-1874-49a0-acc3-f6a033028a37" (UID: "66b8be12-1874-49a0-acc3-f6a033028a37"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:10.692828 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.692677 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:10.692828 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.692722 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7c6k\" (UniqueName: \"kubernetes.io/projected/66b8be12-1874-49a0-acc3-f6a033028a37-kube-api-access-p7c6k\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:10.692828 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:10.692739 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b8be12-1874-49a0-acc3-f6a033028a37-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:11.157599 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:11.157270 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" event={"ID":"7e61d857-3dc0-4120-89d1-a9e531947e51","Type":"ContainerDied","Data":"d323b7f118d3b66acc258473dacfcf9031286b0b1a7ddb6becc98cf87896c94e"} Oct 31 03:35:11.157599 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:11.157334 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d323b7f118d3b66acc258473dacfcf9031286b0b1a7ddb6becc98cf87896c94e" Oct 31 03:35:11.157599 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:11.157435 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9z27" Oct 31 03:35:11.168423 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:11.167888 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" event={"ID":"d9690ccb-5acc-47a7-a664-1d655523e64e","Type":"ContainerStarted","Data":"0401f539d79faa8dd792adf73f57c2cdf521c6dcd936936b8c31f17bd5872679"} Oct 31 03:35:11.171910 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:11.171873 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" event={"ID":"66b8be12-1874-49a0-acc3-f6a033028a37","Type":"ContainerDied","Data":"e86cbec6404972fe359b866f7ff2037113ddc5252595aacaa4c294ebef242198"} Oct 31 03:35:11.172171 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:11.172105 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e86cbec6404972fe359b866f7ff2037113ddc5252595aacaa4c294ebef242198" Oct 31 03:35:11.172234 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:11.172182 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb5hks" Oct 31 03:35:11.287195 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:11.283227 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" podStartSLOduration=2.770553074 podStartE2EDuration="51.283207009s" totalImagesPullingTime="48.512653935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:11.280552798 +0000 UTC m=+408.281178029" watchObservedRunningTime="2026-10-31 03:35:11.283207009 +0000 UTC m=+408.283832239" Oct 31 03:35:12.179977 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:12.179943 4426 generic.go:356] "Generic (PLEG): container finished" podID="6c1351f6-f856-42e2-b702-8dbf4b943f5f" containerID="94c645a9653a3e335cd9547163342ced20d7629a4bddb5a851ce46b8565753c9" exitCode=0 Oct 31 03:35:12.180401 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:12.180013 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" event={"ID":"6c1351f6-f856-42e2-b702-8dbf4b943f5f","Type":"ContainerDied","Data":"94c645a9653a3e335cd9547163342ced20d7629a4bddb5a851ce46b8565753c9"} Oct 31 03:35:12.182095 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:12.182069 4426 generic.go:356] "Generic (PLEG): container finished" podID="c63067d6-40c5-4473-aa7f-dfcb1b52d141" containerID="8757041c23369416667aeba3dc4c30021280537c9d82513abcbf38d0b7c9fc9b" exitCode=0 Oct 31 03:35:12.182234 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:12.182158 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" event={"ID":"c63067d6-40c5-4473-aa7f-dfcb1b52d141","Type":"ContainerDied","Data":"8757041c23369416667aeba3dc4c30021280537c9d82513abcbf38d0b7c9fc9b"} Oct 31 03:35:12.184096 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:12.184072 4426 generic.go:356] "Generic (PLEG): container finished" podID="5b323f7a-fc47-494d-a832-ee7096a5cc85" containerID="1941c998e08035694941fc8b31f5854f987b2721778139c9353e688d16b0464f" exitCode=0 Oct 31 03:35:12.184217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:12.184148 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" event={"ID":"5b323f7a-fc47-494d-a832-ee7096a5cc85","Type":"ContainerDied","Data":"1941c998e08035694941fc8b31f5854f987b2721778139c9353e688d16b0464f"} Oct 31 03:35:12.186258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:12.186233 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" event={"ID":"919ff02d-ecd8-4a04-92fb-92f539b4c676","Type":"ContainerStarted","Data":"f07186346c9d1ec9b20348c4c635770713bdfa5b2ed30f2c69993f322d4c2eef"} Oct 31 03:35:12.188261 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:12.188239 4426 generic.go:356] "Generic (PLEG): container finished" podID="d9690ccb-5acc-47a7-a664-1d655523e64e" containerID="0401f539d79faa8dd792adf73f57c2cdf521c6dcd936936b8c31f17bd5872679" exitCode=0 Oct 31 03:35:12.188353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:12.188273 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" event={"ID":"d9690ccb-5acc-47a7-a664-1d655523e64e","Type":"ContainerDied","Data":"0401f539d79faa8dd792adf73f57c2cdf521c6dcd936936b8c31f17bd5872679"} Oct 31 03:35:12.402849 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:12.402803 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" podStartSLOduration=2.795224789 podStartE2EDuration="52.402787904s" totalImagesPullingTime="49.607563115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:12.401745258 +0000 UTC m=+409.402370486" watchObservedRunningTime="2026-10-31 03:35:12.402787904 +0000 UTC m=+409.403413152" Oct 31 03:35:13.194277 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.194243 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" event={"ID":"6a8607f3-58c0-42db-8f69-5a17b8d22b93","Type":"ContainerStarted","Data":"df14cf15e35eb00a37bcd092d82a999e0ff5d80fa95a900cbbdaa22d79f134d9"} Oct 31 03:35:13.196302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.196278 4426 generic.go:356] "Generic (PLEG): container finished" podID="919ff02d-ecd8-4a04-92fb-92f539b4c676" containerID="f07186346c9d1ec9b20348c4c635770713bdfa5b2ed30f2c69993f322d4c2eef" exitCode=0 Oct 31 03:35:13.196429 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.196348 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" event={"ID":"919ff02d-ecd8-4a04-92fb-92f539b4c676","Type":"ContainerDied","Data":"f07186346c9d1ec9b20348c4c635770713bdfa5b2ed30f2c69993f322d4c2eef"} Oct 31 03:35:13.198394 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.198373 4426 generic.go:356] "Generic (PLEG): container finished" podID="bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" containerID="223592f4ec258926cc9e2ead04da829e23f40f5dc85c7ee4b3d07d6f3d63178f" exitCode=0 Oct 31 03:35:13.198488 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.198453 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" event={"ID":"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636","Type":"ContainerDied","Data":"223592f4ec258926cc9e2ead04da829e23f40f5dc85c7ee4b3d07d6f3d63178f"} Oct 31 03:35:13.200047 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.200029 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:35:13.200770 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.200750 4426 generic.go:356] "Generic (PLEG): container finished" podID="ccff26f6-b677-45ee-920a-1692c44f4fe9" containerID="272682bdfe8134426085b5c6e4a477116d719718628dceccdeabd5c727faecf9" exitCode=1 Oct 31 03:35:13.200840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.200822 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" event={"ID":"ccff26f6-b677-45ee-920a-1692c44f4fe9","Type":"ContainerDied","Data":"272682bdfe8134426085b5c6e4a477116d719718628dceccdeabd5c727faecf9"} Oct 31 03:35:13.202909 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.202882 4426 generic.go:356] "Generic (PLEG): container finished" podID="7221fc5a-653b-45cc-bd81-3f87a9816a90" containerID="2f0abf421da7653bf3c4b3002db12312055799d5df98ff8fa15c1eb8cd294cba" exitCode=0 Oct 31 03:35:13.202994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.202941 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" event={"ID":"7221fc5a-653b-45cc-bd81-3f87a9816a90","Type":"ContainerDied","Data":"2f0abf421da7653bf3c4b3002db12312055799d5df98ff8fa15c1eb8cd294cba"} Oct 31 03:35:13.382469 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.382434 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:35:13.414730 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.414707 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:35:13.417804 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.417784 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:35:13.421200 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.421181 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:35:13.519341 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519268 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-bundle\") pod \"d9690ccb-5acc-47a7-a664-1d655523e64e\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " Oct 31 03:35:13.519341 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519300 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-util\" (UniqueName: \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-util\") pod \"d9690ccb-5acc-47a7-a664-1d655523e64e\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " Oct 31 03:35:13.519341 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519326 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-util\" (UniqueName: \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-util\") pod \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " Oct 31 03:35:13.519583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519362 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b323f7a-fc47-494d-a832-ee7096a5cc85-kube-api-access-wq5cm\" (UniqueName: \"kubernetes.io/projected/5b323f7a-fc47-494d-a832-ee7096a5cc85-kube-api-access-wq5cm\") pod \"5b323f7a-fc47-494d-a832-ee7096a5cc85\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " Oct 31 03:35:13.519583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519392 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-bundle\") pod \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " Oct 31 03:35:13.519583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519440 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c1351f6-f856-42e2-b702-8dbf4b943f5f-kube-api-access-h8twj\" (UniqueName: \"kubernetes.io/projected/6c1351f6-f856-42e2-b702-8dbf4b943f5f-kube-api-access-h8twj\") pod \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " Oct 31 03:35:13.519583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519477 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-util\" (UniqueName: \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-util\") pod \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " Oct 31 03:35:13.519583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519526 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9690ccb-5acc-47a7-a664-1d655523e64e-kube-api-access-lhf5w\" (UniqueName: \"kubernetes.io/projected/d9690ccb-5acc-47a7-a664-1d655523e64e-kube-api-access-lhf5w\") pod \"d9690ccb-5acc-47a7-a664-1d655523e64e\" (UID: \"d9690ccb-5acc-47a7-a664-1d655523e64e\") " Oct 31 03:35:13.519840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519592 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-bundle\") pod \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\" (UID: \"6c1351f6-f856-42e2-b702-8dbf4b943f5f\") " Oct 31 03:35:13.519840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519627 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63067d6-40c5-4473-aa7f-dfcb1b52d141-kube-api-access-gb7kb\" (UniqueName: \"kubernetes.io/projected/c63067d6-40c5-4473-aa7f-dfcb1b52d141-kube-api-access-gb7kb\") pod \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\" (UID: \"c63067d6-40c5-4473-aa7f-dfcb1b52d141\") " Oct 31 03:35:13.519840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519680 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-bundle\") pod \"5b323f7a-fc47-494d-a832-ee7096a5cc85\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " Oct 31 03:35:13.519840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519711 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-util\" (UniqueName: \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-util\") pod \"5b323f7a-fc47-494d-a832-ee7096a5cc85\" (UID: \"5b323f7a-fc47-494d-a832-ee7096a5cc85\") " Oct 31 03:35:13.520023 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.519836 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-bundle" pod "d9690ccb-5acc-47a7-a664-1d655523e64e" (UID: "d9690ccb-5acc-47a7-a664-1d655523e64e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:13.520081 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.520041 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.520603 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.520450 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-bundle" pod "c63067d6-40c5-4473-aa7f-dfcb1b52d141" (UID: "c63067d6-40c5-4473-aa7f-dfcb1b52d141"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:13.520736 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.520665 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-bundle" pod "6c1351f6-f856-42e2-b702-8dbf4b943f5f" (UID: "6c1351f6-f856-42e2-b702-8dbf4b943f5f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:13.520963 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.520944 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-bundle" pod "5b323f7a-fc47-494d-a832-ee7096a5cc85" (UID: "5b323f7a-fc47-494d-a832-ee7096a5cc85"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:13.523202 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.523150 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63067d6-40c5-4473-aa7f-dfcb1b52d141-kube-api-access-gb7kb" pod "c63067d6-40c5-4473-aa7f-dfcb1b52d141" (UID: "c63067d6-40c5-4473-aa7f-dfcb1b52d141"). InnerVolumeSpecName "kube-api-access-gb7kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:13.523316 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.523244 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1351f6-f856-42e2-b702-8dbf4b943f5f-kube-api-access-h8twj" pod "6c1351f6-f856-42e2-b702-8dbf4b943f5f" (UID: "6c1351f6-f856-42e2-b702-8dbf4b943f5f"). InnerVolumeSpecName "kube-api-access-h8twj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:13.523474 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.523435 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b323f7a-fc47-494d-a832-ee7096a5cc85-kube-api-access-wq5cm" pod "5b323f7a-fc47-494d-a832-ee7096a5cc85" (UID: "5b323f7a-fc47-494d-a832-ee7096a5cc85"). InnerVolumeSpecName "kube-api-access-wq5cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:13.523474 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.523452 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9690ccb-5acc-47a7-a664-1d655523e64e-kube-api-access-lhf5w" pod "d9690ccb-5acc-47a7-a664-1d655523e64e" (UID: "d9690ccb-5acc-47a7-a664-1d655523e64e"). InnerVolumeSpecName "kube-api-access-lhf5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:13.527788 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.527763 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-util" pod "6c1351f6-f856-42e2-b702-8dbf4b943f5f" (UID: "6c1351f6-f856-42e2-b702-8dbf4b943f5f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:13.528453 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.528431 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-util" pod "d9690ccb-5acc-47a7-a664-1d655523e64e" (UID: "d9690ccb-5acc-47a7-a664-1d655523e64e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:13.528557 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.528538 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-util" pod "c63067d6-40c5-4473-aa7f-dfcb1b52d141" (UID: "c63067d6-40c5-4473-aa7f-dfcb1b52d141"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:13.529197 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.529183 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-util" pod "5b323f7a-fc47-494d-a832-ee7096a5cc85" (UID: "5b323f7a-fc47-494d-a832-ee7096a5cc85"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:13.620796 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.620764 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7kb\" (UniqueName: \"kubernetes.io/projected/c63067d6-40c5-4473-aa7f-dfcb1b52d141-kube-api-access-gb7kb\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.620796 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.620794 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.620970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.620805 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b323f7a-fc47-494d-a832-ee7096a5cc85-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.620970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.620814 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9690ccb-5acc-47a7-a664-1d655523e64e-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.620970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.620821 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.620970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.620829 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq5cm\" (UniqueName: \"kubernetes.io/projected/5b323f7a-fc47-494d-a832-ee7096a5cc85-kube-api-access-wq5cm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.620970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.620838 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.620970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.620847 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8twj\" (UniqueName: \"kubernetes.io/projected/6c1351f6-f856-42e2-b702-8dbf4b943f5f-kube-api-access-h8twj\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.620970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.620854 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c63067d6-40c5-4473-aa7f-dfcb1b52d141-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.620970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.620862 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhf5w\" (UniqueName: \"kubernetes.io/projected/d9690ccb-5acc-47a7-a664-1d655523e64e-kube-api-access-lhf5w\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.620970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.620871 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c1351f6-f856-42e2-b702-8dbf4b943f5f-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:13.659981 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:13.659925 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" podStartSLOduration=4.004680031 podStartE2EDuration="45.659905453s" totalImagesPullingTime="41.655225422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:13.659351873 +0000 UTC m=+410.659977100" watchObservedRunningTime="2026-10-31 03:35:13.659905453 +0000 UTC m=+410.660530681" Oct 31 03:35:14.210116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.210084 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" Oct 31 03:35:14.210116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.210096 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vzxj8" event={"ID":"d9690ccb-5acc-47a7-a664-1d655523e64e","Type":"ContainerDied","Data":"fe8473080b6af26ebfe6cd91b52fbf2ae69e1625a568f262bbb83d840996cf0a"} Oct 31 03:35:14.210562 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.210156 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe8473080b6af26ebfe6cd91b52fbf2ae69e1625a568f262bbb83d840996cf0a" Oct 31 03:35:14.212157 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.212113 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" Oct 31 03:35:14.212157 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.212151 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tssc8" event={"ID":"6c1351f6-f856-42e2-b702-8dbf4b943f5f","Type":"ContainerDied","Data":"427c2a24a986f36d0659a1b91842ef9713e9ecc017d52a2e0dba7645f219daba"} Oct 31 03:35:14.212377 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.212190 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="427c2a24a986f36d0659a1b91842ef9713e9ecc017d52a2e0dba7645f219daba" Oct 31 03:35:14.214172 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.214146 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" event={"ID":"c63067d6-40c5-4473-aa7f-dfcb1b52d141","Type":"ContainerDied","Data":"9317b0e911bed12b5e59fff8ee0d78f91aaec745f863d77db459f3be4942a9c8"} Oct 31 03:35:14.214172 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.214161 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gdg7t" Oct 31 03:35:14.214343 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.214187 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9317b0e911bed12b5e59fff8ee0d78f91aaec745f863d77db459f3be4942a9c8" Oct 31 03:35:14.216984 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.216958 4426 generic.go:356] "Generic (PLEG): container finished" podID="8230184b-11d2-41bc-9725-69c1b9bd4b63" containerID="2abdadd396603bacdea9b8af36f37e48773f178a90901166ef8f76debc941d67" exitCode=0 Oct 31 03:35:14.217163 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.217042 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" event={"ID":"8230184b-11d2-41bc-9725-69c1b9bd4b63","Type":"ContainerDied","Data":"2abdadd396603bacdea9b8af36f37e48773f178a90901166ef8f76debc941d67"} Oct 31 03:35:14.221470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.221445 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" Oct 31 03:35:14.221611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.221585 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctljb" event={"ID":"5b323f7a-fc47-494d-a832-ee7096a5cc85","Type":"ContainerDied","Data":"ecf518f4f09a23471fadaf5f756d1cc9c69a9fb69badfa8b77c193fac0eac6a8"} Oct 31 03:35:14.221669 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.221630 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecf518f4f09a23471fadaf5f756d1cc9c69a9fb69badfa8b77c193fac0eac6a8" Oct 31 03:35:14.240593 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.226189 4426 generic.go:356] "Generic (PLEG): container finished" podID="6a8607f3-58c0-42db-8f69-5a17b8d22b93" containerID="df14cf15e35eb00a37bcd092d82a999e0ff5d80fa95a900cbbdaa22d79f134d9" exitCode=0 Oct 31 03:35:14.240593 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.226371 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" event={"ID":"6a8607f3-58c0-42db-8f69-5a17b8d22b93","Type":"ContainerDied","Data":"df14cf15e35eb00a37bcd092d82a999e0ff5d80fa95a900cbbdaa22d79f134d9"} Oct 31 03:35:14.370966 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.370927 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:35:14.374022 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.373995 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:35:14.458207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.458181 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:35:14.461670 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.461603 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:35:14.466014 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.465989 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:35:14.527161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.527111 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccff26f6-b677-45ee-920a-1692c44f4fe9-kube-api-access-kr6k7\" (UniqueName: \"kubernetes.io/projected/ccff26f6-b677-45ee-920a-1692c44f4fe9-kube-api-access-kr6k7\") pod \"ccff26f6-b677-45ee-920a-1692c44f4fe9\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " Oct 31 03:35:14.527346 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.527170 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-util\" (UniqueName: \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-util\") pod \"ccff26f6-b677-45ee-920a-1692c44f4fe9\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " Oct 31 03:35:14.527346 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.527223 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-bundle\") pod \"ccff26f6-b677-45ee-920a-1692c44f4fe9\" (UID: \"ccff26f6-b677-45ee-920a-1692c44f4fe9\") " Oct 31 03:35:14.527891 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.527867 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-bundle" pod "ccff26f6-b677-45ee-920a-1692c44f4fe9" (UID: "ccff26f6-b677-45ee-920a-1692c44f4fe9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:14.529336 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.529301 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccff26f6-b677-45ee-920a-1692c44f4fe9-kube-api-access-kr6k7" pod "ccff26f6-b677-45ee-920a-1692c44f4fe9" (UID: "ccff26f6-b677-45ee-920a-1692c44f4fe9"). InnerVolumeSpecName "kube-api-access-kr6k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:14.532981 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.532958 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-util" pod "ccff26f6-b677-45ee-920a-1692c44f4fe9" (UID: "ccff26f6-b677-45ee-920a-1692c44f4fe9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:14.628562 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.628512 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-bundle\") pod \"7221fc5a-653b-45cc-bd81-3f87a9816a90\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " Oct 31 03:35:14.628728 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.628582 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-bundle\") pod \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " Oct 31 03:35:14.628728 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.628607 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7221fc5a-653b-45cc-bd81-3f87a9816a90-kube-api-access-f2tw8\" (UniqueName: \"kubernetes.io/projected/7221fc5a-653b-45cc-bd81-3f87a9816a90-kube-api-access-f2tw8\") pod \"7221fc5a-653b-45cc-bd81-3f87a9816a90\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " Oct 31 03:35:14.628728 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.628635 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-kube-api-access-ft586\" (UniqueName: \"kubernetes.io/projected/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-kube-api-access-ft586\") pod \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " Oct 31 03:35:14.628728 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.628662 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-bundle\" (UniqueName: \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-bundle\") pod \"919ff02d-ecd8-4a04-92fb-92f539b4c676\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " Oct 31 03:35:14.628728 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.628685 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-util\" (UniqueName: \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-util\") pod \"919ff02d-ecd8-4a04-92fb-92f539b4c676\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " Oct 31 03:35:14.628994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.628758 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-util\" (UniqueName: \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-util\") pod \"7221fc5a-653b-45cc-bd81-3f87a9816a90\" (UID: \"7221fc5a-653b-45cc-bd81-3f87a9816a90\") " Oct 31 03:35:14.628994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.628831 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-util\" (UniqueName: \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-util\") pod \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\" (UID: \"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636\") " Oct 31 03:35:14.628994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.628864 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919ff02d-ecd8-4a04-92fb-92f539b4c676-kube-api-access-nrhmc\" (UniqueName: \"kubernetes.io/projected/919ff02d-ecd8-4a04-92fb-92f539b4c676-kube-api-access-nrhmc\") pod \"919ff02d-ecd8-4a04-92fb-92f539b4c676\" (UID: \"919ff02d-ecd8-4a04-92fb-92f539b4c676\") " Oct 31 03:35:14.629168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.629104 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6k7\" (UniqueName: \"kubernetes.io/projected/ccff26f6-b677-45ee-920a-1692c44f4fe9-kube-api-access-kr6k7\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:14.629168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.629149 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:14.629168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.629164 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccff26f6-b677-45ee-920a-1692c44f4fe9-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:14.629404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.629376 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-bundle" pod "7221fc5a-653b-45cc-bd81-3f87a9816a90" (UID: "7221fc5a-653b-45cc-bd81-3f87a9816a90"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:14.629485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.629412 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-bundle" pod "919ff02d-ecd8-4a04-92fb-92f539b4c676" (UID: "919ff02d-ecd8-4a04-92fb-92f539b4c676"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:14.629539 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.629475 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-bundle" pod "bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" (UID: "bf0c8575-d3a4-4b4b-9cc9-d64f60c51636"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:14.637211 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.637154 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-util" pod "919ff02d-ecd8-4a04-92fb-92f539b4c676" (UID: "919ff02d-ecd8-4a04-92fb-92f539b4c676"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:14.637614 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.637585 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7221fc5a-653b-45cc-bd81-3f87a9816a90-kube-api-access-f2tw8" pod "7221fc5a-653b-45cc-bd81-3f87a9816a90" (UID: "7221fc5a-653b-45cc-bd81-3f87a9816a90"). InnerVolumeSpecName "kube-api-access-f2tw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:14.637614 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.637595 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-kube-api-access-ft586" pod "bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" (UID: "bf0c8575-d3a4-4b4b-9cc9-d64f60c51636"). InnerVolumeSpecName "kube-api-access-ft586". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:14.637614 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.637607 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919ff02d-ecd8-4a04-92fb-92f539b4c676-kube-api-access-nrhmc" pod "919ff02d-ecd8-4a04-92fb-92f539b4c676" (UID: "919ff02d-ecd8-4a04-92fb-92f539b4c676"). InnerVolumeSpecName "kube-api-access-nrhmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:14.637821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.637799 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-util" pod "bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" (UID: "bf0c8575-d3a4-4b4b-9cc9-d64f60c51636"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:14.638026 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.638006 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-util" pod "7221fc5a-653b-45cc-bd81-3f87a9816a90" (UID: "7221fc5a-653b-45cc-bd81-3f87a9816a90"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:14.730190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.730075 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:14.730190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.730107 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2tw8\" (UniqueName: \"kubernetes.io/projected/7221fc5a-653b-45cc-bd81-3f87a9816a90-kube-api-access-f2tw8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:14.730190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.730142 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft586\" (UniqueName: \"kubernetes.io/projected/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-kube-api-access-ft586\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:14.730190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.730155 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:14.730190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.730166 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/919ff02d-ecd8-4a04-92fb-92f539b4c676-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:14.730190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.730177 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:14.730190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.730188 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0c8575-d3a4-4b4b-9cc9-d64f60c51636-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:14.730503 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.730199 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrhmc\" (UniqueName: \"kubernetes.io/projected/919ff02d-ecd8-4a04-92fb-92f539b4c676-kube-api-access-nrhmc\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:14.730503 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:14.730211 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7221fc5a-653b-45cc-bd81-3f87a9816a90-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:15.234218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.234190 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" Oct 31 03:35:15.234683 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.234219 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhq97" event={"ID":"7221fc5a-653b-45cc-bd81-3f87a9816a90","Type":"ContainerDied","Data":"e45ec7b2fa1293171438806fe313757599d96f034113920e16338b4001f57476"} Oct 31 03:35:15.234683 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.234271 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e45ec7b2fa1293171438806fe313757599d96f034113920e16338b4001f57476" Oct 31 03:35:15.236350 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.236324 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" event={"ID":"919ff02d-ecd8-4a04-92fb-92f539b4c676","Type":"ContainerDied","Data":"94f27305b9b682447c8d0d2be175b3294bfff92d9aebbc2b8562af0bfe90ab36"} Oct 31 03:35:15.236545 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.236526 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94f27305b9b682447c8d0d2be175b3294bfff92d9aebbc2b8562af0bfe90ab36" Oct 31 03:35:15.236545 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.236377 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wvlf" Oct 31 03:35:15.238656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.238600 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" event={"ID":"bf0c8575-d3a4-4b4b-9cc9-d64f60c51636","Type":"ContainerDied","Data":"4c24e7cb85bfceba1da0f9c62db869d4c9b41b6eec4f9ee5670759ee3508945f"} Oct 31 03:35:15.238656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.238600 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5m52s" Oct 31 03:35:15.238656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.238637 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c24e7cb85bfceba1da0f9c62db869d4c9b41b6eec4f9ee5670759ee3508945f" Oct 31 03:35:15.240034 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.240016 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:35:15.240812 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.240793 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" event={"ID":"ccff26f6-b677-45ee-920a-1692c44f4fe9","Type":"ContainerDied","Data":"a081d6fce197db9a1c4d0c919c41d2f9007b952114a786735cd805db28271d1f"} Oct 31 03:35:15.240880 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.240822 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a081d6fce197db9a1c4d0c919c41d2f9007b952114a786735cd805db28271d1f" Oct 31 03:35:15.240943 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.240932 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc" Oct 31 03:35:15.371564 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.371537 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:35:15.404057 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.404033 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:35:15.535704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.535665 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-bundle\") pod \"8230184b-11d2-41bc-9725-69c1b9bd4b63\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " Oct 31 03:35:15.535893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.535721 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8230184b-11d2-41bc-9725-69c1b9bd4b63-kube-api-access-6x8xq\" (UniqueName: \"kubernetes.io/projected/8230184b-11d2-41bc-9725-69c1b9bd4b63-kube-api-access-6x8xq\") pod \"8230184b-11d2-41bc-9725-69c1b9bd4b63\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " Oct 31 03:35:15.535893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.535795 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-bundle\") pod \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " Oct 31 03:35:15.535893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.535817 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-util\" (UniqueName: \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-util\") pod \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " Oct 31 03:35:15.535893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.535843 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-util\" (UniqueName: \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-util\") pod \"8230184b-11d2-41bc-9725-69c1b9bd4b63\" (UID: \"8230184b-11d2-41bc-9725-69c1b9bd4b63\") " Oct 31 03:35:15.535893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.535872 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a8607f3-58c0-42db-8f69-5a17b8d22b93-kube-api-access-wlvtw\" (UniqueName: \"kubernetes.io/projected/6a8607f3-58c0-42db-8f69-5a17b8d22b93-kube-api-access-wlvtw\") pod \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\" (UID: \"6a8607f3-58c0-42db-8f69-5a17b8d22b93\") " Oct 31 03:35:15.536287 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.536254 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-bundle" pod "8230184b-11d2-41bc-9725-69c1b9bd4b63" (UID: "8230184b-11d2-41bc-9725-69c1b9bd4b63"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:15.536413 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.536385 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-bundle" pod "6a8607f3-58c0-42db-8f69-5a17b8d22b93" (UID: "6a8607f3-58c0-42db-8f69-5a17b8d22b93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:15.537981 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.537949 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8230184b-11d2-41bc-9725-69c1b9bd4b63-kube-api-access-6x8xq" pod "8230184b-11d2-41bc-9725-69c1b9bd4b63" (UID: "8230184b-11d2-41bc-9725-69c1b9bd4b63"). InnerVolumeSpecName "kube-api-access-6x8xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:15.537981 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.537963 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8607f3-58c0-42db-8f69-5a17b8d22b93-kube-api-access-wlvtw" pod "6a8607f3-58c0-42db-8f69-5a17b8d22b93" (UID: "6a8607f3-58c0-42db-8f69-5a17b8d22b93"). InnerVolumeSpecName "kube-api-access-wlvtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:15.542555 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.542534 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-util" pod "6a8607f3-58c0-42db-8f69-5a17b8d22b93" (UID: "6a8607f3-58c0-42db-8f69-5a17b8d22b93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:15.543853 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.543832 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-util" pod "8230184b-11d2-41bc-9725-69c1b9bd4b63" (UID: "8230184b-11d2-41bc-9725-69c1b9bd4b63"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:35:15.637191 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.637156 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:15.637191 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.637185 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a8607f3-58c0-42db-8f69-5a17b8d22b93-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:15.637191 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.637194 4426 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-util\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:15.637417 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.637203 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvtw\" (UniqueName: \"kubernetes.io/projected/6a8607f3-58c0-42db-8f69-5a17b8d22b93-kube-api-access-wlvtw\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:15.637417 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.637214 4426 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8230184b-11d2-41bc-9725-69c1b9bd4b63-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:15.637417 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:15.637223 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x8xq\" (UniqueName: \"kubernetes.io/projected/8230184b-11d2-41bc-9725-69c1b9bd4b63-kube-api-access-6x8xq\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:16.247273 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:16.247240 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" Oct 31 03:35:16.247674 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:16.247234 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66wfv6" event={"ID":"8230184b-11d2-41bc-9725-69c1b9bd4b63","Type":"ContainerDied","Data":"7cbb4c5b8efcdaaba0c2958749aca3733aa42616c75d404801ab80ec13ec356c"} Oct 31 03:35:16.247674 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:16.247396 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cbb4c5b8efcdaaba0c2958749aca3733aa42616c75d404801ab80ec13ec356c" Oct 31 03:35:16.249419 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:16.249398 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" Oct 31 03:35:16.249562 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:16.249399 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cth9b2" event={"ID":"6a8607f3-58c0-42db-8f69-5a17b8d22b93","Type":"ContainerDied","Data":"6e8fe4042f4fe10ea452ad03a673a72efb3d3cd488b64af917ac47de1f596d13"} Oct 31 03:35:16.249562 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:16.249510 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e8fe4042f4fe10ea452ad03a673a72efb3d3cd488b64af917ac47de1f596d13" Oct 31 03:35:17.431812 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.431775 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh"] Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432036 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5b323f7a-fc47-494d-a832-ee7096a5cc85" containerName="pull" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432048 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b323f7a-fc47-494d-a832-ee7096a5cc85" containerName="pull" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432060 4426 cpu_manager.go:414] "containerMap: removing container" podUID="8230184b-11d2-41bc-9725-69c1b9bd4b63" containerName="extract" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432065 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8230184b-11d2-41bc-9725-69c1b9bd4b63" containerName="extract" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432073 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c63067d6-40c5-4473-aa7f-dfcb1b52d141" containerName="util" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432078 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63067d6-40c5-4473-aa7f-dfcb1b52d141" containerName="util" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432086 4426 cpu_manager.go:414] "containerMap: removing container" podUID="d9690ccb-5acc-47a7-a664-1d655523e64e" containerName="extract" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432091 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9690ccb-5acc-47a7-a664-1d655523e64e" containerName="extract" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432097 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6a8607f3-58c0-42db-8f69-5a17b8d22b93" containerName="extract" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432102 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a8607f3-58c0-42db-8f69-5a17b8d22b93" containerName="extract" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432108 4426 cpu_manager.go:414] "containerMap: removing container" podUID="d9690ccb-5acc-47a7-a664-1d655523e64e" containerName="util" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432113 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9690ccb-5acc-47a7-a664-1d655523e64e" containerName="util" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432141 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5b323f7a-fc47-494d-a832-ee7096a5cc85" containerName="util" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432149 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b323f7a-fc47-494d-a832-ee7096a5cc85" containerName="util" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432157 4426 cpu_manager.go:414] "containerMap: removing container" podUID="8230184b-11d2-41bc-9725-69c1b9bd4b63" containerName="pull" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432164 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8230184b-11d2-41bc-9725-69c1b9bd4b63" containerName="pull" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432172 4426 cpu_manager.go:414] "containerMap: removing container" podUID="bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" containerName="pull" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432179 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" containerName="pull" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432186 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6a8607f3-58c0-42db-8f69-5a17b8d22b93" containerName="pull" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432190 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a8607f3-58c0-42db-8f69-5a17b8d22b93" containerName="pull" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432196 4426 cpu_manager.go:414] "containerMap: removing container" podUID="66b8be12-1874-49a0-acc3-f6a033028a37" containerName="extract" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432200 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b8be12-1874-49a0-acc3-f6a033028a37" containerName="extract" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432207 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6a8607f3-58c0-42db-8f69-5a17b8d22b93" containerName="util" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432211 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a8607f3-58c0-42db-8f69-5a17b8d22b93" containerName="util" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432216 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6c1351f6-f856-42e2-b702-8dbf4b943f5f" containerName="pull" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432220 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1351f6-f856-42e2-b702-8dbf4b943f5f" containerName="pull" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432228 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ccff26f6-b677-45ee-920a-1692c44f4fe9" containerName="pull" Oct 31 03:35:17.432217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432232 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccff26f6-b677-45ee-920a-1692c44f4fe9" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432238 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7e61d857-3dc0-4120-89d1-a9e531947e51" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432243 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e61d857-3dc0-4120-89d1-a9e531947e51" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432250 4426 cpu_manager.go:414] "containerMap: removing container" podUID="d9690ccb-5acc-47a7-a664-1d655523e64e" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432254 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9690ccb-5acc-47a7-a664-1d655523e64e" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432259 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7221fc5a-653b-45cc-bd81-3f87a9816a90" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432263 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7221fc5a-653b-45cc-bd81-3f87a9816a90" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432269 4426 cpu_manager.go:414] "containerMap: removing container" podUID="919ff02d-ecd8-4a04-92fb-92f539b4c676" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432274 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919ff02d-ecd8-4a04-92fb-92f539b4c676" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432278 4426 cpu_manager.go:414] "containerMap: removing container" podUID="66b8be12-1874-49a0-acc3-f6a033028a37" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432282 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b8be12-1874-49a0-acc3-f6a033028a37" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432292 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ccff26f6-b677-45ee-920a-1692c44f4fe9" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432296 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccff26f6-b677-45ee-920a-1692c44f4fe9" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432302 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7221fc5a-653b-45cc-bd81-3f87a9816a90" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432305 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7221fc5a-653b-45cc-bd81-3f87a9816a90" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432310 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7e61d857-3dc0-4120-89d1-a9e531947e51" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432315 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e61d857-3dc0-4120-89d1-a9e531947e51" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432321 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6c1351f6-f856-42e2-b702-8dbf4b943f5f" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432325 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1351f6-f856-42e2-b702-8dbf4b943f5f" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432331 4426 cpu_manager.go:414] "containerMap: removing container" podUID="919ff02d-ecd8-4a04-92fb-92f539b4c676" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432335 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919ff02d-ecd8-4a04-92fb-92f539b4c676" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432344 4426 cpu_manager.go:414] "containerMap: removing container" podUID="bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432349 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432354 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7221fc5a-653b-45cc-bd81-3f87a9816a90" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432358 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7221fc5a-653b-45cc-bd81-3f87a9816a90" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432365 4426 cpu_manager.go:414] "containerMap: removing container" podUID="bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432369 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432375 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c63067d6-40c5-4473-aa7f-dfcb1b52d141" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432380 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63067d6-40c5-4473-aa7f-dfcb1b52d141" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432385 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ccff26f6-b677-45ee-920a-1692c44f4fe9" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432389 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccff26f6-b677-45ee-920a-1692c44f4fe9" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432395 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6c1351f6-f856-42e2-b702-8dbf4b943f5f" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432399 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1351f6-f856-42e2-b702-8dbf4b943f5f" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432404 4426 cpu_manager.go:414] "containerMap: removing container" podUID="8230184b-11d2-41bc-9725-69c1b9bd4b63" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432409 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8230184b-11d2-41bc-9725-69c1b9bd4b63" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432414 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c63067d6-40c5-4473-aa7f-dfcb1b52d141" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432418 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63067d6-40c5-4473-aa7f-dfcb1b52d141" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432424 4426 cpu_manager.go:414] "containerMap: removing container" podUID="66b8be12-1874-49a0-acc3-f6a033028a37" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432429 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b8be12-1874-49a0-acc3-f6a033028a37" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432434 4426 cpu_manager.go:414] "containerMap: removing container" podUID="919ff02d-ecd8-4a04-92fb-92f539b4c676" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432437 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919ff02d-ecd8-4a04-92fb-92f539b4c676" containerName="pull" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432443 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5b323f7a-fc47-494d-a832-ee7096a5cc85" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432448 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b323f7a-fc47-494d-a832-ee7096a5cc85" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432456 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7e61d857-3dc0-4120-89d1-a9e531947e51" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432459 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e61d857-3dc0-4120-89d1-a9e531947e51" containerName="util" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432505 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9690ccb-5acc-47a7-a664-1d655523e64e" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432513 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a8607f3-58c0-42db-8f69-5a17b8d22b93" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432519 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b8be12-1874-49a0-acc3-f6a033028a37" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432525 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e61d857-3dc0-4120-89d1-a9e531947e51" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432532 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1351f6-f856-42e2-b702-8dbf4b943f5f" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432537 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="919ff02d-ecd8-4a04-92fb-92f539b4c676" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432543 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf0c8575-d3a4-4b4b-9cc9-d64f60c51636" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432548 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="7221fc5a-653b-45cc-bd81-3f87a9816a90" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432553 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63067d6-40c5-4473-aa7f-dfcb1b52d141" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432558 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccff26f6-b677-45ee-920a-1692c44f4fe9" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432563 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b323f7a-fc47-494d-a832-ee7096a5cc85" containerName="extract" Oct 31 03:35:17.432994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.432568 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="8230184b-11d2-41bc-9725-69c1b9bd4b63" containerName="extract" Oct 31 03:35:17.436594 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.436577 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh" Oct 31 03:35:17.487050 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.487021 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh"] Oct 31 03:35:17.531134 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.531102 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:35:17.531313 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.531212 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 03:35:17.531361 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.531312 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zwxxd\"" Oct 31 03:35:17.550212 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.550185 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38a2ca8c-5af0-49d6-9b25-b547142b6ae0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh\" (UID: \"38a2ca8c-5af0-49d6-9b25-b547142b6ae0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh" Oct 31 03:35:17.550269 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.550217 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slqld\" (UniqueName: \"kubernetes.io/projected/38a2ca8c-5af0-49d6-9b25-b547142b6ae0-kube-api-access-slqld\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh\" (UID: \"38a2ca8c-5af0-49d6-9b25-b547142b6ae0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh" Oct 31 03:35:17.651559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.651519 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38a2ca8c-5af0-49d6-9b25-b547142b6ae0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh\" (UID: \"38a2ca8c-5af0-49d6-9b25-b547142b6ae0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh" Oct 31 03:35:17.651559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.651561 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slqld\" (UniqueName: \"kubernetes.io/projected/38a2ca8c-5af0-49d6-9b25-b547142b6ae0-kube-api-access-slqld\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh\" (UID: \"38a2ca8c-5af0-49d6-9b25-b547142b6ae0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh" Oct 31 03:35:17.651936 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.651914 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38a2ca8c-5af0-49d6-9b25-b547142b6ae0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh\" (UID: \"38a2ca8c-5af0-49d6-9b25-b547142b6ae0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh" Oct 31 03:35:17.709379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.709311 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slqld\" (UniqueName: \"kubernetes.io/projected/38a2ca8c-5af0-49d6-9b25-b547142b6ae0-kube-api-access-slqld\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh\" (UID: \"38a2ca8c-5af0-49d6-9b25-b547142b6ae0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh" Oct 31 03:35:17.747971 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:17.747940 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh" Oct 31 03:35:18.162644 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:18.162617 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh"] Oct 31 03:35:18.165138 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:35:18.165096 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a2ca8c_5af0_49d6_9b25_b547142b6ae0.slice/crio-043259b211048a2fff36fc39c4e9cc5b4d37141fb0502176475a7c2f216bc508 WatchSource:0}: Error finding container 043259b211048a2fff36fc39c4e9cc5b4d37141fb0502176475a7c2f216bc508: Status 404 returned error can't find the container with id 043259b211048a2fff36fc39c4e9cc5b4d37141fb0502176475a7c2f216bc508 Oct 31 03:35:18.260966 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:18.260930 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh" event={"ID":"38a2ca8c-5af0-49d6-9b25-b547142b6ae0","Type":"ContainerStarted","Data":"043259b211048a2fff36fc39c4e9cc5b4d37141fb0502176475a7c2f216bc508"} Oct 31 03:35:21.276815 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:21.276771 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh" event={"ID":"38a2ca8c-5af0-49d6-9b25-b547142b6ae0","Type":"ContainerStarted","Data":"f1b325d5a17dea039ee3c6d4f89223b2ec8a1b8721a0cc6098f5e7f0411c47b3"} Oct 31 03:35:21.490095 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:21.490034 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-wfcmh" podStartSLOduration=1.534972711 podStartE2EDuration="4.490019786s" totalImagesPullingTime="2.955047075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:21.441780949 +0000 UTC m=+418.442406178" watchObservedRunningTime="2026-10-31 03:35:21.490019786 +0000 UTC m=+418.490645014" Oct 31 03:35:23.138995 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:23.138964 4426 scope.go:122] "RemoveContainer" containerID="7861833dddccd59bc60270491bde6231533a30990a5d10874928513dd24f7490" Oct 31 03:35:28.919028 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:28.918988 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2"] Oct 31 03:35:28.924341 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:28.924320 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.020663 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.020629 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 03:35:29.020663 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.020629 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 03:35:29.020910 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.020776 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 03:35:29.020910 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.020782 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 03:35:29.020910 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.020807 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-lljsf\"" Oct 31 03:35:29.020910 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.020840 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 03:35:29.037683 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.037651 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2"] Oct 31 03:35:29.039746 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.039724 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2238925-dcc1-456f-97fa-238a1d64674f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.039794 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.039762 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2238925-dcc1-456f-97fa-238a1d64674f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.039837 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.039791 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2238925-dcc1-456f-97fa-238a1d64674f-tmp\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.039927 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.039904 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2238925-dcc1-456f-97fa-238a1d64674f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.039988 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.039956 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcz26\" (UniqueName: \"kubernetes.io/projected/b2238925-dcc1-456f-97fa-238a1d64674f-kube-api-access-gcz26\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.040082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.040063 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2238925-dcc1-456f-97fa-238a1d64674f-manifests\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.141457 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.141423 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2238925-dcc1-456f-97fa-238a1d64674f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.141662 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.141464 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2238925-dcc1-456f-97fa-238a1d64674f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.141662 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.141483 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2238925-dcc1-456f-97fa-238a1d64674f-tmp\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.141662 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.141606 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2238925-dcc1-456f-97fa-238a1d64674f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.141662 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.141654 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcz26\" (UniqueName: \"kubernetes.io/projected/b2238925-dcc1-456f-97fa-238a1d64674f-kube-api-access-gcz26\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.141902 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.141702 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2238925-dcc1-456f-97fa-238a1d64674f-manifests\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.142077 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.142041 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2238925-dcc1-456f-97fa-238a1d64674f-manifests\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.144108 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.144083 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2238925-dcc1-456f-97fa-238a1d64674f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.144231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.144155 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2238925-dcc1-456f-97fa-238a1d64674f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.144231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.144213 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2238925-dcc1-456f-97fa-238a1d64674f-tmp\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.144383 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.144367 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2238925-dcc1-456f-97fa-238a1d64674f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.204768 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.204691 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcz26\" (UniqueName: \"kubernetes.io/projected/b2238925-dcc1-456f-97fa-238a1d64674f-kube-api-access-gcz26\") pod \"opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2\" (UID: \"b2238925-dcc1-456f-97fa-238a1d64674f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.239180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.239145 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:29.721332 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:29.721306 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2"] Oct 31 03:35:29.721882 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:35:29.721856 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2238925_dcc1_456f_97fa_238a1d64674f.slice/crio-e920d14a71b16a849defa1244e9030fee96a2a4f39112763baa8ce4ad74afc2d WatchSource:0}: Error finding container e920d14a71b16a849defa1244e9030fee96a2a4f39112763baa8ce4ad74afc2d: Status 404 returned error can't find the container with id e920d14a71b16a849defa1244e9030fee96a2a4f39112763baa8ce4ad74afc2d Oct 31 03:35:30.320479 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:30.320445 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" event={"ID":"b2238925-dcc1-456f-97fa-238a1d64674f","Type":"ContainerStarted","Data":"e920d14a71b16a849defa1244e9030fee96a2a4f39112763baa8ce4ad74afc2d"} Oct 31 03:35:32.848826 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:35:32.848783 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:35:33.805296 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:33.805205 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f"] Oct 31 03:35:33.811513 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:33.811483 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f" Oct 31 03:35:33.861885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:33.861846 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 03:35:33.861885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:33.861845 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-tjsdg\"" Oct 31 03:35:33.861885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:33.861846 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 03:35:33.863985 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:33.863958 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f"] Oct 31 03:35:33.879455 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:33.879418 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a63f1cec-5647-464e-b551-0107f1dcb77e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5bn2f\" (UID: \"a63f1cec-5647-464e-b551-0107f1dcb77e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f" Oct 31 03:35:33.879455 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:33.879456 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdq4z\" (UniqueName: \"kubernetes.io/projected/a63f1cec-5647-464e-b551-0107f1dcb77e-kube-api-access-gdq4z\") pod \"cert-manager-cainjector-5c57bbd4fc-5bn2f\" (UID: \"a63f1cec-5647-464e-b551-0107f1dcb77e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f" Oct 31 03:35:33.980110 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:33.980072 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a63f1cec-5647-464e-b551-0107f1dcb77e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5bn2f\" (UID: \"a63f1cec-5647-464e-b551-0107f1dcb77e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f" Oct 31 03:35:33.980110 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:33.980109 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdq4z\" (UniqueName: \"kubernetes.io/projected/a63f1cec-5647-464e-b551-0107f1dcb77e-kube-api-access-gdq4z\") pod \"cert-manager-cainjector-5c57bbd4fc-5bn2f\" (UID: \"a63f1cec-5647-464e-b551-0107f1dcb77e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f" Oct 31 03:35:34.037840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:34.037800 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a63f1cec-5647-464e-b551-0107f1dcb77e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5bn2f\" (UID: \"a63f1cec-5647-464e-b551-0107f1dcb77e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f" Oct 31 03:35:34.038016 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:34.037963 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdq4z\" (UniqueName: \"kubernetes.io/projected/a63f1cec-5647-464e-b551-0107f1dcb77e-kube-api-access-gdq4z\") pod \"cert-manager-cainjector-5c57bbd4fc-5bn2f\" (UID: \"a63f1cec-5647-464e-b551-0107f1dcb77e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f" Oct 31 03:35:34.132395 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:34.132303 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f" Oct 31 03:35:34.341536 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:34.341499 4426 generic.go:356] "Generic (PLEG): container finished" podID="b2238925-dcc1-456f-97fa-238a1d64674f" containerID="468d97c2f8a903047f6713bb5c60b5cd6685a75baafcec595c17fea5f0084934" exitCode=0 Oct 31 03:35:34.341681 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:34.341578 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" event={"ID":"b2238925-dcc1-456f-97fa-238a1d64674f","Type":"ContainerDied","Data":"468d97c2f8a903047f6713bb5c60b5cd6685a75baafcec595c17fea5f0084934"} Oct 31 03:35:34.496015 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:34.495983 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f"] Oct 31 03:35:34.497902 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:35:34.497870 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda63f1cec_5647_464e_b551_0107f1dcb77e.slice/crio-ae5932da113bcf669439d47941cba41f44826b5dd7e184cd9cb8c5c1753d44c7 WatchSource:0}: Error finding container ae5932da113bcf669439d47941cba41f44826b5dd7e184cd9cb8c5c1753d44c7: Status 404 returned error can't find the container with id ae5932da113bcf669439d47941cba41f44826b5dd7e184cd9cb8c5c1753d44c7 Oct 31 03:35:35.352839 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.352802 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f" event={"ID":"a63f1cec-5647-464e-b551-0107f1dcb77e","Type":"ContainerStarted","Data":"ae5932da113bcf669439d47941cba41f44826b5dd7e184cd9cb8c5c1753d44c7"} Oct 31 03:35:35.414540 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.414512 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j"] Oct 31 03:35:35.459113 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.459084 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j" Oct 31 03:35:35.475502 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.475477 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j"] Oct 31 03:35:35.492087 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.492049 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjpz\" (UniqueName: \"kubernetes.io/projected/60f7506b-536a-4604-952d-d7447f9a3787-kube-api-access-zkjpz\") pod \"dns-operator-controller-manager-8557cbdb84-8gg9j\" (UID: \"60f7506b-536a-4604-952d-d7447f9a3787\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j" Oct 31 03:35:35.519543 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.519505 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 03:35:35.519543 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.519505 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 03:35:35.519543 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.519519 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-dpjsw\"" Oct 31 03:35:35.519839 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.519520 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 03:35:35.592667 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.592630 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjpz\" (UniqueName: \"kubernetes.io/projected/60f7506b-536a-4604-952d-d7447f9a3787-kube-api-access-zkjpz\") pod \"dns-operator-controller-manager-8557cbdb84-8gg9j\" (UID: \"60f7506b-536a-4604-952d-d7447f9a3787\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j" Oct 31 03:35:35.654481 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.654443 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkjpz\" (UniqueName: \"kubernetes.io/projected/60f7506b-536a-4604-952d-d7447f9a3787-kube-api-access-zkjpz\") pod \"dns-operator-controller-manager-8557cbdb84-8gg9j\" (UID: \"60f7506b-536a-4604-952d-d7447f9a3787\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j" Oct 31 03:35:35.789324 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:35.789284 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j" Oct 31 03:35:36.178496 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:36.178473 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j"] Oct 31 03:35:36.180542 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:35:36.180513 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f7506b_536a_4604_952d_d7447f9a3787.slice/crio-054d9dd87e9086798b6b484fb5cb74e83786d74932e8d2c78094683e3bcc1af8 WatchSource:0}: Error finding container 054d9dd87e9086798b6b484fb5cb74e83786d74932e8d2c78094683e3bcc1af8: Status 404 returned error can't find the container with id 054d9dd87e9086798b6b484fb5cb74e83786d74932e8d2c78094683e3bcc1af8 Oct 31 03:35:36.359533 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:36.359494 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" event={"ID":"b2238925-dcc1-456f-97fa-238a1d64674f","Type":"ContainerStarted","Data":"5185e08af96954ea2f93ab8bfe2fff56972831082c6c5ca31099beebd1bd6815"} Oct 31 03:35:36.359913 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:36.359574 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:36.361170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:36.361142 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j" event={"ID":"60f7506b-536a-4604-952d-d7447f9a3787","Type":"ContainerStarted","Data":"054d9dd87e9086798b6b484fb5cb74e83786d74932e8d2c78094683e3bcc1af8"} Oct 31 03:35:36.594021 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:36.593875 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" podStartSLOduration=3.406168471 podStartE2EDuration="8.593856684s" totalImagesPullingTime="4.72570242s" totalInitContainerRuntime="461.985793ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:36.546782037 +0000 UTC m=+433.547407268" watchObservedRunningTime="2026-10-31 03:35:36.593856684 +0000 UTC m=+433.594481912" Oct 31 03:35:39.383023 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:39.382961 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f" event={"ID":"a63f1cec-5647-464e-b551-0107f1dcb77e","Type":"ContainerStarted","Data":"9473d9ae57831e8aeee339c40949aef59447084ac06e11719415668a631fe428"} Oct 31 03:35:39.493959 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:39.491949 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5bn2f" podStartSLOduration=2.487107392 podStartE2EDuration="6.491930724s" totalImagesPullingTime="4.004823332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:39.491117431 +0000 UTC m=+436.491742651" watchObservedRunningTime="2026-10-31 03:35:39.491930724 +0000 UTC m=+436.492555952" Oct 31 03:35:40.389732 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:40.389694 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j" event={"ID":"60f7506b-536a-4604-952d-d7447f9a3787","Type":"ContainerStarted","Data":"5c1c549ac48ba87eecf9d8c4a7f6872a89b095a2ea59d3b9e4c4a0331df8ec1c"} Oct 31 03:35:40.500751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:40.500690 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j" podStartSLOduration=1.806344629 podStartE2EDuration="5.500675032s" totalImagesPullingTime="3.694330403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:40.499965311 +0000 UTC m=+437.500590538" watchObservedRunningTime="2026-10-31 03:35:40.500675032 +0000 UTC m=+437.501300259" Oct 31 03:35:41.394361 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:41.394324 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j" Oct 31 03:35:42.113251 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.113208 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m"] Oct 31 03:35:42.116664 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.116646 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 03:35:42.149092 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.149053 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfp8\" (UniqueName: \"kubernetes.io/projected/63813f18-28ec-4dab-bd59-17d1a825c4a8-kube-api-access-kkfp8\") pod \"kuadrant-operator-controller-manager-678d87766f-4zz6m\" (UID: \"63813f18-28ec-4dab-bd59-17d1a825c4a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 03:35:42.149302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.149101 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63813f18-28ec-4dab-bd59-17d1a825c4a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4zz6m\" (UID: \"63813f18-28ec-4dab-bd59-17d1a825c4a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 03:35:42.165925 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.165900 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tpf45\"" Oct 31 03:35:42.214608 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.214568 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m"] Oct 31 03:35:42.250441 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.250401 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfp8\" (UniqueName: \"kubernetes.io/projected/63813f18-28ec-4dab-bd59-17d1a825c4a8-kube-api-access-kkfp8\") pod \"kuadrant-operator-controller-manager-678d87766f-4zz6m\" (UID: \"63813f18-28ec-4dab-bd59-17d1a825c4a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 03:35:42.250635 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.250453 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63813f18-28ec-4dab-bd59-17d1a825c4a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4zz6m\" (UID: \"63813f18-28ec-4dab-bd59-17d1a825c4a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 03:35:42.250887 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.250869 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63813f18-28ec-4dab-bd59-17d1a825c4a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4zz6m\" (UID: \"63813f18-28ec-4dab-bd59-17d1a825c4a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 03:35:42.309088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.309054 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfp8\" (UniqueName: \"kubernetes.io/projected/63813f18-28ec-4dab-bd59-17d1a825c4a8-kube-api-access-kkfp8\") pod \"kuadrant-operator-controller-manager-678d87766f-4zz6m\" (UID: \"63813f18-28ec-4dab-bd59-17d1a825c4a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 03:35:42.428961 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.428918 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 03:35:42.892426 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:42.892388 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m"] Oct 31 03:35:42.926217 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:35:42.926174 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63813f18_28ec_4dab_bd59_17d1a825c4a8.slice/crio-d9f60b22eb221f4c1a5ab6f5e1b0983354773401e33966160fc595a5f6eaad5e WatchSource:0}: Error finding container d9f60b22eb221f4c1a5ab6f5e1b0983354773401e33966160fc595a5f6eaad5e: Status 404 returned error can't find the container with id d9f60b22eb221f4c1a5ab6f5e1b0983354773401e33966160fc595a5f6eaad5e Oct 31 03:35:43.404542 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:43.404507 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" event={"ID":"63813f18-28ec-4dab-bd59-17d1a825c4a8","Type":"ContainerStarted","Data":"d9f60b22eb221f4c1a5ab6f5e1b0983354773401e33966160fc595a5f6eaad5e"} Oct 31 03:35:45.197808 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:45.197765 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf"] Oct 31 03:35:45.201499 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:45.201473 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf" Oct 31 03:35:45.257759 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:45.257725 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-frqnl\"" Oct 31 03:35:45.276365 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:45.276204 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhvr\" (UniqueName: \"kubernetes.io/projected/c79d294e-f5fc-4f02-bdda-5aa7885145bf-kube-api-access-qjhvr\") pod \"authorino-operator-7b7bb4bc77-gc4kf\" (UID: \"c79d294e-f5fc-4f02-bdda-5aa7885145bf\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf" Oct 31 03:35:45.306297 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:45.306252 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf"] Oct 31 03:35:45.377790 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:45.377742 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhvr\" (UniqueName: \"kubernetes.io/projected/c79d294e-f5fc-4f02-bdda-5aa7885145bf-kube-api-access-qjhvr\") pod \"authorino-operator-7b7bb4bc77-gc4kf\" (UID: \"c79d294e-f5fc-4f02-bdda-5aa7885145bf\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf" Oct 31 03:35:45.439186 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:45.439153 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhvr\" (UniqueName: \"kubernetes.io/projected/c79d294e-f5fc-4f02-bdda-5aa7885145bf-kube-api-access-qjhvr\") pod \"authorino-operator-7b7bb4bc77-gc4kf\" (UID: \"c79d294e-f5fc-4f02-bdda-5aa7885145bf\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf" Oct 31 03:35:45.518136 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:45.518037 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf" Oct 31 03:35:45.896719 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:45.896658 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf"] Oct 31 03:35:45.899211 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:35:45.899178 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79d294e_f5fc_4f02_bdda_5aa7885145bf.slice/crio-41d2c40afa19eb406b6d020e68172a5c69732b52471d42d798c6a7715b568d34 WatchSource:0}: Error finding container 41d2c40afa19eb406b6d020e68172a5c69732b52471d42d798c6a7715b568d34: Status 404 returned error can't find the container with id 41d2c40afa19eb406b6d020e68172a5c69732b52471d42d798c6a7715b568d34 Oct 31 03:35:46.420255 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:46.420215 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf" event={"ID":"c79d294e-f5fc-4f02-bdda-5aa7885145bf","Type":"ContainerStarted","Data":"41d2c40afa19eb406b6d020e68172a5c69732b52471d42d798c6a7715b568d34"} Oct 31 03:35:47.369430 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:47.369392 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dcdbf6b7b-szsh2" Oct 31 03:35:48.059695 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.059656 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn"] Oct 31 03:35:48.063550 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.063521 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn" Oct 31 03:35:48.104619 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.104571 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnw6h\" (UniqueName: \"kubernetes.io/projected/7a80e800-84ce-434c-98f6-f0e8823ed5d7-kube-api-access-hnw6h\") pod \"limitador-operator-controller-manager-5778bb4449-2dljn\" (UID: \"7a80e800-84ce-434c-98f6-f0e8823ed5d7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn" Oct 31 03:35:48.112674 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.112634 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pkmbj\"" Oct 31 03:35:48.120031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.119945 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn"] Oct 31 03:35:48.205382 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.205340 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnw6h\" (UniqueName: \"kubernetes.io/projected/7a80e800-84ce-434c-98f6-f0e8823ed5d7-kube-api-access-hnw6h\") pod \"limitador-operator-controller-manager-5778bb4449-2dljn\" (UID: \"7a80e800-84ce-434c-98f6-f0e8823ed5d7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn" Oct 31 03:35:48.264324 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.264291 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnw6h\" (UniqueName: \"kubernetes.io/projected/7a80e800-84ce-434c-98f6-f0e8823ed5d7-kube-api-access-hnw6h\") pod \"limitador-operator-controller-manager-5778bb4449-2dljn\" (UID: \"7a80e800-84ce-434c-98f6-f0e8823ed5d7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn" Oct 31 03:35:48.380831 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.380792 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn" Oct 31 03:35:48.436933 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.436892 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf" event={"ID":"c79d294e-f5fc-4f02-bdda-5aa7885145bf","Type":"ContainerStarted","Data":"30732e31f861db50f329832d2c6b7cc3967a5a27d6b420f39f0c2d048f10b882"} Oct 31 03:35:48.437119 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.437075 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf" Oct 31 03:35:48.547250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.547191 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf" podStartSLOduration=1.212409336 podStartE2EDuration="3.547173191s" totalImagesPullingTime="2.334763855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:48.545665961 +0000 UTC m=+445.546291193" watchObservedRunningTime="2026-10-31 03:35:48.547173191 +0000 UTC m=+445.547798420" Oct 31 03:35:48.754969 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:48.754934 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn"] Oct 31 03:35:48.759233 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:35:48.759186 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a80e800_84ce_434c_98f6_f0e8823ed5d7.slice/crio-4170c157d2b85ee02b929b5faa14168036991044652217dcbc7bac836e4bcf16 WatchSource:0}: Error finding container 4170c157d2b85ee02b929b5faa14168036991044652217dcbc7bac836e4bcf16: Status 404 returned error can't find the container with id 4170c157d2b85ee02b929b5faa14168036991044652217dcbc7bac836e4bcf16 Oct 31 03:35:49.443390 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:49.443350 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn" event={"ID":"7a80e800-84ce-434c-98f6-f0e8823ed5d7","Type":"ContainerStarted","Data":"4170c157d2b85ee02b929b5faa14168036991044652217dcbc7bac836e4bcf16"} Oct 31 03:35:49.445021 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:49.444984 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" event={"ID":"63813f18-28ec-4dab-bd59-17d1a825c4a8","Type":"ContainerStarted","Data":"0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6"} Oct 31 03:35:49.445192 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:49.445168 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 03:35:49.559391 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:49.559326 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" podStartSLOduration=1.402209913 podStartE2EDuration="7.559310866s" totalImagesPullingTime="6.157100953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:49.557828738 +0000 UTC m=+446.558453962" watchObservedRunningTime="2026-10-31 03:35:49.559310866 +0000 UTC m=+446.559936094" Oct 31 03:35:52.400971 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:52.400938 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8gg9j" Oct 31 03:35:52.461196 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:52.461157 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn" event={"ID":"7a80e800-84ce-434c-98f6-f0e8823ed5d7","Type":"ContainerStarted","Data":"93b2a680620880261894e51c0e9a39a68f422b8af306644a731cd4acc0c2d9d1"} Oct 31 03:35:52.461366 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:52.461300 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn" Oct 31 03:35:52.622289 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:52.622235 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn" podStartSLOduration=1.9786740090000001 podStartE2EDuration="4.622219475s" totalImagesPullingTime="2.643545466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:52.62083959 +0000 UTC m=+449.621464818" watchObservedRunningTime="2026-10-31 03:35:52.622219475 +0000 UTC m=+449.622844703" Oct 31 03:35:59.447323 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:35:59.447290 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc4kf" Oct 31 03:36:00.452962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:00.452929 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 03:36:02.846236 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:36:02.846198 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:36:03.468257 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.468227 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2dljn" Oct 31 03:36:03.668052 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.668014 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9"] Oct 31 03:36:03.672105 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.672080 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.724817 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.724736 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-smvsr\"" Oct 31 03:36:03.730842 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.730803 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9"] Oct 31 03:36:03.737553 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.737519 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.737690 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.737582 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.737741 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.737690 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.737788 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.737741 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.737859 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.737839 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpq89\" (UniqueName: \"kubernetes.io/projected/16b2427e-bf1e-4892-ad55-9d5f2b93849b-kube-api-access-cpq89\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.737911 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.737898 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.737954 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.737918 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.738056 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.738032 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.738121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.738106 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839247 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839208 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839439 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839261 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839439 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839281 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839439 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839305 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839439 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839332 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpq89\" (UniqueName: \"kubernetes.io/projected/16b2427e-bf1e-4892-ad55-9d5f2b93849b-kube-api-access-cpq89\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839439 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839385 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839439 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839412 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839439 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839439 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839846 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839485 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839846 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839669 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839846 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839721 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.839983 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839850 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.840021 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.839973 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.840252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.840225 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.841805 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.841769 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.842017 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.841997 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.915148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.915087 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b2427e-bf1e-4892-ad55-9d5f2b93849b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.915672 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.915260 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpq89\" (UniqueName: \"kubernetes.io/projected/16b2427e-bf1e-4892-ad55-9d5f2b93849b-kube-api-access-cpq89\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9\" (UID: \"16b2427e-bf1e-4892-ad55-9d5f2b93849b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:03.988851 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:03.988773 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:04.409059 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:04.409035 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9"] Oct 31 03:36:04.411241 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:36:04.411216 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b2427e_bf1e_4892_ad55_9d5f2b93849b.slice/crio-61870cd519af1e0a39c631f5f7501a588524ee74c9e0e9e7a2231d2c95c2c5ab WatchSource:0}: Error finding container 61870cd519af1e0a39c631f5f7501a588524ee74c9e0e9e7a2231d2c95c2c5ab: Status 404 returned error can't find the container with id 61870cd519af1e0a39c631f5f7501a588524ee74c9e0e9e7a2231d2c95c2c5ab Oct 31 03:36:04.413271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:04.413234 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:36:04.413373 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:04.413322 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:36:04.413435 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:04.413373 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:36:04.520968 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:04.520932 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" event={"ID":"16b2427e-bf1e-4892-ad55-9d5f2b93849b","Type":"ContainerStarted","Data":"642d2cee2260df754db0ff77fd1633e820df98f8c693fe40841f9304a0aabdc2"} Oct 31 03:36:04.521119 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:04.520989 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" event={"ID":"16b2427e-bf1e-4892-ad55-9d5f2b93849b","Type":"ContainerStarted","Data":"61870cd519af1e0a39c631f5f7501a588524ee74c9e0e9e7a2231d2c95c2c5ab"} Oct 31 03:36:04.634266 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:04.634213 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" podStartSLOduration=1.634198796 podStartE2EDuration="1.634198796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:04.633966666 +0000 UTC m=+461.634591896" watchObservedRunningTime="2026-10-31 03:36:04.634198796 +0000 UTC m=+461.634824027" Oct 31 03:36:04.989246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:04.989211 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:05.993768 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:05.993735 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:06.531333 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:06.531306 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:06.532312 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:06.532292 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9" Oct 31 03:36:09.810141 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:09.810090 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jdxqr"] Oct 31 03:36:09.813736 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:09.813712 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jdxqr" Oct 31 03:36:09.864841 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:09.864792 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ppjlh\"" Oct 31 03:36:09.867024 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:09.866995 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jdxqr"] Oct 31 03:36:09.893903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:09.893860 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb8jm\" (UniqueName: \"kubernetes.io/projected/8688c100-da74-49f6-aa6c-d8894b408a67-kube-api-access-jb8jm\") pod \"authorino-57f7f8659f-jdxqr\" (UID: \"8688c100-da74-49f6-aa6c-d8894b408a67\") " pod="kuadrant-system/authorino-57f7f8659f-jdxqr" Oct 31 03:36:09.994783 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:09.994745 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb8jm\" (UniqueName: \"kubernetes.io/projected/8688c100-da74-49f6-aa6c-d8894b408a67-kube-api-access-jb8jm\") pod \"authorino-57f7f8659f-jdxqr\" (UID: \"8688c100-da74-49f6-aa6c-d8894b408a67\") " pod="kuadrant-system/authorino-57f7f8659f-jdxqr" Oct 31 03:36:10.052794 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:10.052759 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb8jm\" (UniqueName: \"kubernetes.io/projected/8688c100-da74-49f6-aa6c-d8894b408a67-kube-api-access-jb8jm\") pod \"authorino-57f7f8659f-jdxqr\" (UID: \"8688c100-da74-49f6-aa6c-d8894b408a67\") " pod="kuadrant-system/authorino-57f7f8659f-jdxqr" Oct 31 03:36:10.125834 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:10.125753 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jdxqr" Oct 31 03:36:10.494469 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:10.494444 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jdxqr"] Oct 31 03:36:10.496409 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:36:10.496378 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8688c100_da74_49f6_aa6c_d8894b408a67.slice/crio-63e019698d5a11d38c2280fec67ede85bab1fc9c035877f0e4a8eb061b02e0c9 WatchSource:0}: Error finding container 63e019698d5a11d38c2280fec67ede85bab1fc9c035877f0e4a8eb061b02e0c9: Status 404 returned error can't find the container with id 63e019698d5a11d38c2280fec67ede85bab1fc9c035877f0e4a8eb061b02e0c9 Oct 31 03:36:10.552368 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:10.552334 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jdxqr" event={"ID":"8688c100-da74-49f6-aa6c-d8894b408a67","Type":"ContainerStarted","Data":"63e019698d5a11d38c2280fec67ede85bab1fc9c035877f0e4a8eb061b02e0c9"} Oct 31 03:36:12.460861 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.460822 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:36:12.463743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.463716 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" Oct 31 03:36:12.512795 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.512761 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 03:36:12.520394 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.520353 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/155f33ed-b4f9-4914-934d-82b04dcba3a1-config-file\") pod \"limitador-limitador-7c9b87d574-22j29\" (UID: \"155f33ed-b4f9-4914-934d-82b04dcba3a1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" Oct 31 03:36:12.520394 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.520405 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6lvx\" (UniqueName: \"kubernetes.io/projected/155f33ed-b4f9-4914-934d-82b04dcba3a1-kube-api-access-s6lvx\") pod \"limitador-limitador-7c9b87d574-22j29\" (UID: \"155f33ed-b4f9-4914-934d-82b04dcba3a1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" Oct 31 03:36:12.559370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.559333 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kk2nt\"" Oct 31 03:36:12.561619 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.561592 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:36:12.621808 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.621771 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/155f33ed-b4f9-4914-934d-82b04dcba3a1-config-file\") pod \"limitador-limitador-7c9b87d574-22j29\" (UID: \"155f33ed-b4f9-4914-934d-82b04dcba3a1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" Oct 31 03:36:12.621808 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.621812 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6lvx\" (UniqueName: \"kubernetes.io/projected/155f33ed-b4f9-4914-934d-82b04dcba3a1-kube-api-access-s6lvx\") pod \"limitador-limitador-7c9b87d574-22j29\" (UID: \"155f33ed-b4f9-4914-934d-82b04dcba3a1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" Oct 31 03:36:12.622550 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.622526 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/155f33ed-b4f9-4914-934d-82b04dcba3a1-config-file\") pod \"limitador-limitador-7c9b87d574-22j29\" (UID: \"155f33ed-b4f9-4914-934d-82b04dcba3a1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" Oct 31 03:36:12.680507 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.680473 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6lvx\" (UniqueName: \"kubernetes.io/projected/155f33ed-b4f9-4914-934d-82b04dcba3a1-kube-api-access-s6lvx\") pod \"limitador-limitador-7c9b87d574-22j29\" (UID: \"155f33ed-b4f9-4914-934d-82b04dcba3a1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" Oct 31 03:36:12.776660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.776564 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" Oct 31 03:36:12.783257 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:12.783226 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:36:13.146589 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:13.146498 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:36:13.149858 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:36:13.149830 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155f33ed_b4f9_4914_934d_82b04dcba3a1.slice/crio-d1094832c40e7840041896b0824aadc1fda0ebb83e49752c40ee729ae3ea378a WatchSource:0}: Error finding container d1094832c40e7840041896b0824aadc1fda0ebb83e49752c40ee729ae3ea378a: Status 404 returned error can't find the container with id d1094832c40e7840041896b0824aadc1fda0ebb83e49752c40ee729ae3ea378a Oct 31 03:36:13.570567 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:13.570484 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" event={"ID":"155f33ed-b4f9-4914-934d-82b04dcba3a1","Type":"ContainerStarted","Data":"d1094832c40e7840041896b0824aadc1fda0ebb83e49752c40ee729ae3ea378a"} Oct 31 03:36:14.577110 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:14.577073 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jdxqr" event={"ID":"8688c100-da74-49f6-aa6c-d8894b408a67","Type":"ContainerStarted","Data":"2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369"} Oct 31 03:36:14.688241 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:14.688176 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-jdxqr" podStartSLOduration=2.5815700550000003 podStartE2EDuration="5.688161203s" totalImagesPullingTime="3.106591148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:14.687626372 +0000 UTC m=+471.688251599" watchObservedRunningTime="2026-10-31 03:36:14.688161203 +0000 UTC m=+471.688786430" Oct 31 03:36:15.583734 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:15.583637 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" event={"ID":"155f33ed-b4f9-4914-934d-82b04dcba3a1","Type":"ContainerStarted","Data":"531a76cb3093dad301a82ab387f7a5c78626890ff7679dc0a3aa8cefca537c3b"} Oct 31 03:36:15.584140 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:15.583862 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" Oct 31 03:36:15.694285 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:15.694231 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" podStartSLOduration=1.6503021869999999 podStartE2EDuration="3.694217968s" totalImagesPullingTime="2.043915781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:15.692343515 +0000 UTC m=+472.692968743" watchObservedRunningTime="2026-10-31 03:36:15.694217968 +0000 UTC m=+472.694843195" Oct 31 03:36:18.316954 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.316920 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jdxqr"] Oct 31 03:36:18.317380 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.317075 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-jdxqr" podUID="8688c100-da74-49f6-aa6c-d8894b408a67" containerName="authorino" containerID="cri-o://2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369" gracePeriod=30 Oct 31 03:36:18.566671 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.566642 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jdxqr" Oct 31 03:36:18.598624 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.598547 4426 generic.go:356] "Generic (PLEG): container finished" podID="8688c100-da74-49f6-aa6c-d8894b408a67" containerID="2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369" exitCode=0 Oct 31 03:36:18.598624 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.598586 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jdxqr" Oct 31 03:36:18.598781 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.598637 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jdxqr" event={"ID":"8688c100-da74-49f6-aa6c-d8894b408a67","Type":"ContainerDied","Data":"2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369"} Oct 31 03:36:18.598781 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.598682 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jdxqr" event={"ID":"8688c100-da74-49f6-aa6c-d8894b408a67","Type":"ContainerDied","Data":"63e019698d5a11d38c2280fec67ede85bab1fc9c035877f0e4a8eb061b02e0c9"} Oct 31 03:36:18.598781 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.598712 4426 scope.go:122] "RemoveContainer" containerID="2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369" Oct 31 03:36:18.607426 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.607411 4426 scope.go:122] "RemoveContainer" containerID="2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369" Oct 31 03:36:18.607717 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:36:18.607701 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369\": container with ID starting with 2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369 not found: ID does not exist" containerID="2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369" Oct 31 03:36:18.607756 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.607726 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369"} err="failed to get container status \"2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369\": rpc error: code = NotFound desc = could not find container \"2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369\": container with ID starting with 2b99bf6c1b42fb3c9d92854596385d3518929bf6921afecc85b329528a7d2369 not found: ID does not exist" Oct 31 03:36:18.680631 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.680604 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8688c100-da74-49f6-aa6c-d8894b408a67-kube-api-access-jb8jm\" (UniqueName: \"kubernetes.io/projected/8688c100-da74-49f6-aa6c-d8894b408a67-kube-api-access-jb8jm\") pod \"8688c100-da74-49f6-aa6c-d8894b408a67\" (UID: \"8688c100-da74-49f6-aa6c-d8894b408a67\") " Oct 31 03:36:18.682997 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.682957 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8688c100-da74-49f6-aa6c-d8894b408a67-kube-api-access-jb8jm" pod "8688c100-da74-49f6-aa6c-d8894b408a67" (UID: "8688c100-da74-49f6-aa6c-d8894b408a67"). InnerVolumeSpecName "kube-api-access-jb8jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:18.781308 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:18.781267 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb8jm\" (UniqueName: \"kubernetes.io/projected/8688c100-da74-49f6-aa6c-d8894b408a67-kube-api-access-jb8jm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:19.011306 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:19.011271 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jdxqr"] Oct 31 03:36:19.014823 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:19.014791 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jdxqr"] Oct 31 03:36:19.126870 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:19.126833 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8688c100-da74-49f6-aa6c-d8894b408a67" path="/var/lib/kubelet/pods/8688c100-da74-49f6-aa6c-d8894b408a67/volumes" Oct 31 03:36:26.589394 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:36:26.589361 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-22j29" Oct 31 03:36:32.848740 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:36:32.848699 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:37:02.845098 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:37:02.845054 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:37:06.807573 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.807538 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-x2mcc"] Oct 31 03:37:06.808042 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.808020 4426 cpu_manager.go:414] "containerMap: removing container" podUID="8688c100-da74-49f6-aa6c-d8894b408a67" containerName="authorino" Oct 31 03:37:06.808097 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.808044 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8688c100-da74-49f6-aa6c-d8894b408a67" containerName="authorino" Oct 31 03:37:06.808170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.808146 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="8688c100-da74-49f6-aa6c-d8894b408a67" containerName="authorino" Oct 31 03:37:06.813321 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.813304 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:06.871955 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.871917 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-webhook-cert\") pod \"maas-controller-579cf7c6d-x2mcc\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:06.872160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.871964 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-metrics-certs\") pod \"maas-controller-579cf7c6d-x2mcc\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:06.872160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.871996 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbx6m\" (UniqueName: \"kubernetes.io/projected/696975b8-8ef4-4192-9b87-25f10dc6d138-kube-api-access-gbx6m\") pod \"maas-controller-579cf7c6d-x2mcc\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:06.909107 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.909076 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 03:37:06.909450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.909379 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 03:37:06.909450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.909389 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 03:37:06.909450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.909418 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 03:37:06.909450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.909422 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 03:37:06.909450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.909419 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8xpcq\"" Oct 31 03:37:06.913379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.913329 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-x2mcc"] Oct 31 03:37:06.972472 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.972433 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-webhook-cert\") pod \"maas-controller-579cf7c6d-x2mcc\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:06.972673 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.972481 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-metrics-certs\") pod \"maas-controller-579cf7c6d-x2mcc\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:06.972673 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.972515 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbx6m\" (UniqueName: \"kubernetes.io/projected/696975b8-8ef4-4192-9b87-25f10dc6d138-kube-api-access-gbx6m\") pod \"maas-controller-579cf7c6d-x2mcc\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:06.975235 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.975200 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-webhook-cert\") pod \"maas-controller-579cf7c6d-x2mcc\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:06.975373 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:06.975308 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-metrics-certs\") pod \"maas-controller-579cf7c6d-x2mcc\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:07.030435 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.030394 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbx6m\" (UniqueName: \"kubernetes.io/projected/696975b8-8ef4-4192-9b87-25f10dc6d138-kube-api-access-gbx6m\") pod \"maas-controller-579cf7c6d-x2mcc\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:07.126184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.126067 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:07.232830 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.232793 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-565fd9c896-565x2"] Oct 31 03:37:07.237919 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.237900 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:07.293337 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.293305 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-565fd9c896-565x2"] Oct 31 03:37:07.377328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.377240 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-webhook-cert\") pod \"maas-controller-565fd9c896-565x2\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:07.377328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.377280 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-metrics-certs\") pod \"maas-controller-565fd9c896-565x2\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:07.377328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.377306 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8bl7\" (UniqueName: \"kubernetes.io/projected/2df4691e-00da-49e4-bb70-02ad5fad1625-kube-api-access-b8bl7\") pod \"maas-controller-565fd9c896-565x2\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:07.478656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.478613 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-webhook-cert\") pod \"maas-controller-565fd9c896-565x2\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:07.478656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.478657 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-metrics-certs\") pod \"maas-controller-565fd9c896-565x2\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:07.478864 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.478684 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8bl7\" (UniqueName: \"kubernetes.io/projected/2df4691e-00da-49e4-bb70-02ad5fad1625-kube-api-access-b8bl7\") pod \"maas-controller-565fd9c896-565x2\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:07.481119 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.481092 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-metrics-certs\") pod \"maas-controller-565fd9c896-565x2\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:07.481119 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.481108 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-webhook-cert\") pod \"maas-controller-565fd9c896-565x2\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:07.493549 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.493528 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-x2mcc"] Oct 31 03:37:07.495625 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:37:07.495599 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696975b8_8ef4_4192_9b87_25f10dc6d138.slice/crio-2aeaf86716a93df62b63676256021444ea084c9de94bd10e3878c15dfc883b4e WatchSource:0}: Error finding container 2aeaf86716a93df62b63676256021444ea084c9de94bd10e3878c15dfc883b4e: Status 404 returned error can't find the container with id 2aeaf86716a93df62b63676256021444ea084c9de94bd10e3878c15dfc883b4e Oct 31 03:37:07.536659 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.536629 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8bl7\" (UniqueName: \"kubernetes.io/projected/2df4691e-00da-49e4-bb70-02ad5fad1625-kube-api-access-b8bl7\") pod \"maas-controller-565fd9c896-565x2\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:07.551432 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.551408 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:07.856097 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.856060 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" event={"ID":"696975b8-8ef4-4192-9b87-25f10dc6d138","Type":"ContainerStarted","Data":"2aeaf86716a93df62b63676256021444ea084c9de94bd10e3878c15dfc883b4e"} Oct 31 03:37:07.923100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:07.923075 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-565fd9c896-565x2"] Oct 31 03:37:07.924914 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:37:07.924888 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df4691e_00da_49e4_bb70_02ad5fad1625.slice/crio-85a7eacb3ada90e029b94b412f52226af0d2b7578bcf5eacd34f68bad3f52708 WatchSource:0}: Error finding container 85a7eacb3ada90e029b94b412f52226af0d2b7578bcf5eacd34f68bad3f52708: Status 404 returned error can't find the container with id 85a7eacb3ada90e029b94b412f52226af0d2b7578bcf5eacd34f68bad3f52708 Oct 31 03:37:08.864459 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:08.864417 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-565x2" event={"ID":"2df4691e-00da-49e4-bb70-02ad5fad1625","Type":"ContainerStarted","Data":"85a7eacb3ada90e029b94b412f52226af0d2b7578bcf5eacd34f68bad3f52708"} Oct 31 03:37:11.886617 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:11.886505 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" event={"ID":"696975b8-8ef4-4192-9b87-25f10dc6d138","Type":"ContainerStarted","Data":"45da9d1f12641a3219c8eba2d688d1dfc47231f1245fe935e15aece54facc84a"} Oct 31 03:37:11.887052 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:11.886614 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:11.888254 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:11.888228 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-565x2" event={"ID":"2df4691e-00da-49e4-bb70-02ad5fad1625","Type":"ContainerStarted","Data":"22b50971fa3d01f6d36fd0d59a0d1e422c1cedaba6f9a9b4998c815328c8e5fc"} Oct 31 03:37:11.888353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:11.888292 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:11.999068 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:11.999011 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" podStartSLOduration=2.468949328 podStartE2EDuration="5.998995198s" totalImagesPullingTime="3.53004587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:11.997140606 +0000 UTC m=+528.997765825" watchObservedRunningTime="2026-10-31 03:37:11.998995198 +0000 UTC m=+528.999620426" Oct 31 03:37:12.110973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:12.110903 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-565fd9c896-565x2" podStartSLOduration=1.641679954 podStartE2EDuration="5.110882725s" totalImagesPullingTime="3.469202771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:12.109774541 +0000 UTC m=+529.110399771" watchObservedRunningTime="2026-10-31 03:37:12.110882725 +0000 UTC m=+529.111507955" Oct 31 03:37:14.436999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.436963 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6"] Oct 31 03:37:14.440563 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.440545 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:14.489334 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.489307 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 03:37:14.489334 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.489308 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 03:37:14.489581 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.489564 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 03:37:14.536407 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.536361 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-ffpjg\"" Oct 31 03:37:14.538836 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.538804 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6"] Oct 31 03:37:14.540636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.540610 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c14f5021-0344-46ff-a0ad-85b011c7ade3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:14.540743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.540676 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c14f5021-0344-46ff-a0ad-85b011c7ade3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:14.540829 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.540812 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djrcq\" (UniqueName: \"kubernetes.io/projected/c14f5021-0344-46ff-a0ad-85b011c7ade3-kube-api-access-djrcq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:14.540885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.540872 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c14f5021-0344-46ff-a0ad-85b011c7ade3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:14.641316 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.641287 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c14f5021-0344-46ff-a0ad-85b011c7ade3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:14.641498 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.641336 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djrcq\" (UniqueName: \"kubernetes.io/projected/c14f5021-0344-46ff-a0ad-85b011c7ade3-kube-api-access-djrcq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:14.641498 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.641381 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c14f5021-0344-46ff-a0ad-85b011c7ade3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:14.641498 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.641413 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c14f5021-0344-46ff-a0ad-85b011c7ade3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:14.641498 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:37:14.641411 4426 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 31 03:37:14.641713 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:37:14.641523 4426 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c14f5021-0344-46ff-a0ad-85b011c7ade3-proxy-source podName:c14f5021-0344-46ff-a0ad-85b011c7ade3 nodeName:}" failed. No retries permitted until 2026-10-31 03:37:15.141501959 +0000 UTC m=+532.142127168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/c14f5021-0344-46ff-a0ad-85b011c7ade3-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" (UID: "c14f5021-0344-46ff-a0ad-85b011c7ade3") : configmap "usage-tenancy-proxy-source" not found Oct 31 03:37:14.642141 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.642095 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c14f5021-0344-46ff-a0ad-85b011c7ade3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:14.643869 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.643842 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c14f5021-0344-46ff-a0ad-85b011c7ade3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:14.701207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:14.701137 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djrcq\" (UniqueName: \"kubernetes.io/projected/c14f5021-0344-46ff-a0ad-85b011c7ade3-kube-api-access-djrcq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:15.146308 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:15.146279 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c14f5021-0344-46ff-a0ad-85b011c7ade3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:15.147046 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:15.147026 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c14f5021-0344-46ff-a0ad-85b011c7ade3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-l7zq6\" (UID: \"c14f5021-0344-46ff-a0ad-85b011c7ade3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:15.352866 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:15.352823 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:15.721777 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:15.721751 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6"] Oct 31 03:37:15.723456 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:37:15.723429 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14f5021_0344_46ff_a0ad_85b011c7ade3.slice/crio-1bf487248332fc8fa6c1665e74790b4d821e38ea8d1b4672b90a86b47cef4352 WatchSource:0}: Error finding container 1bf487248332fc8fa6c1665e74790b4d821e38ea8d1b4672b90a86b47cef4352: Status 404 returned error can't find the container with id 1bf487248332fc8fa6c1665e74790b4d821e38ea8d1b4672b90a86b47cef4352 Oct 31 03:37:15.910069 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:15.910035 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" event={"ID":"c14f5021-0344-46ff-a0ad-85b011c7ade3","Type":"ContainerStarted","Data":"1bf487248332fc8fa6c1665e74790b4d821e38ea8d1b4672b90a86b47cef4352"} Oct 31 03:37:22.896504 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:22.896466 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:37:22.899637 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:22.899303 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:23.202316 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:23.202239 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-x2mcc"] Oct 31 03:37:23.202693 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:23.202654 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" podUID="696975b8-8ef4-4192-9b87-25f10dc6d138" containerName="manager" containerID="cri-o://45da9d1f12641a3219c8eba2d688d1dfc47231f1245fe935e15aece54facc84a" gracePeriod=30 Oct 31 03:37:23.961317 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:23.961279 4426 generic.go:356] "Generic (PLEG): container finished" podID="696975b8-8ef4-4192-9b87-25f10dc6d138" containerID="45da9d1f12641a3219c8eba2d688d1dfc47231f1245fe935e15aece54facc84a" exitCode=0 Oct 31 03:37:23.961757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:23.961431 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" event={"ID":"696975b8-8ef4-4192-9b87-25f10dc6d138","Type":"ContainerDied","Data":"45da9d1f12641a3219c8eba2d688d1dfc47231f1245fe935e15aece54facc84a"} Oct 31 03:37:23.993016 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:23.992989 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:24.140153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.140098 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-metrics-certs\" (UniqueName: \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-metrics-certs\") pod \"696975b8-8ef4-4192-9b87-25f10dc6d138\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " Oct 31 03:37:24.140324 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.140173 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/696975b8-8ef4-4192-9b87-25f10dc6d138-kube-api-access-gbx6m\" (UniqueName: \"kubernetes.io/projected/696975b8-8ef4-4192-9b87-25f10dc6d138-kube-api-access-gbx6m\") pod \"696975b8-8ef4-4192-9b87-25f10dc6d138\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " Oct 31 03:37:24.140324 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.140247 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-webhook-cert\" (UniqueName: \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-webhook-cert\") pod \"696975b8-8ef4-4192-9b87-25f10dc6d138\" (UID: \"696975b8-8ef4-4192-9b87-25f10dc6d138\") " Oct 31 03:37:24.142635 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.142598 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-webhook-cert" pod "696975b8-8ef4-4192-9b87-25f10dc6d138" (UID: "696975b8-8ef4-4192-9b87-25f10dc6d138"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:37:24.142787 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.142767 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696975b8-8ef4-4192-9b87-25f10dc6d138-kube-api-access-gbx6m" pod "696975b8-8ef4-4192-9b87-25f10dc6d138" (UID: "696975b8-8ef4-4192-9b87-25f10dc6d138"). InnerVolumeSpecName "kube-api-access-gbx6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:24.145469 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.145438 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-metrics-certs" pod "696975b8-8ef4-4192-9b87-25f10dc6d138" (UID: "696975b8-8ef4-4192-9b87-25f10dc6d138"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:37:24.241424 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.241378 4426 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-webhook-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:24.241424 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.241418 4426 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/696975b8-8ef4-4192-9b87-25f10dc6d138-metrics-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:24.241424 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.241432 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbx6m\" (UniqueName: \"kubernetes.io/projected/696975b8-8ef4-4192-9b87-25f10dc6d138-kube-api-access-gbx6m\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:24.969167 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.969134 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" Oct 31 03:37:24.969656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.969167 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-x2mcc" event={"ID":"696975b8-8ef4-4192-9b87-25f10dc6d138","Type":"ContainerDied","Data":"2aeaf86716a93df62b63676256021444ea084c9de94bd10e3878c15dfc883b4e"} Oct 31 03:37:24.969656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:24.969224 4426 scope.go:122] "RemoveContainer" containerID="45da9d1f12641a3219c8eba2d688d1dfc47231f1245fe935e15aece54facc84a" Oct 31 03:37:25.162947 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:25.162911 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-x2mcc"] Oct 31 03:37:25.209343 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:25.209310 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-x2mcc"] Oct 31 03:37:27.130194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:27.130157 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="696975b8-8ef4-4192-9b87-25f10dc6d138" path="/var/lib/kubelet/pods/696975b8-8ef4-4192-9b87-25f10dc6d138/volumes" Oct 31 03:37:30.001191 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:30.001151 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" event={"ID":"c14f5021-0344-46ff-a0ad-85b011c7ade3","Type":"ContainerStarted","Data":"f5939a8818182b92a9f6508d67918ae8094a69a7f93471e16db09af7aed9d029"} Oct 31 03:37:30.001645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:30.001215 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:30.115013 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:30.114698 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" podStartSLOduration=1.980142463 podStartE2EDuration="16.114678352s" totalImagesPullingTime="14.134535889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:30.114174354 +0000 UTC m=+547.114799575" watchObservedRunningTime="2026-10-31 03:37:30.114678352 +0000 UTC m=+547.115303613" Oct 31 03:37:32.845856 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:37:32.845819 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:37:35.401840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.401794 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n4sbl"] Oct 31 03:37:35.402369 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.402342 4426 cpu_manager.go:414] "containerMap: removing container" podUID="696975b8-8ef4-4192-9b87-25f10dc6d138" containerName="manager" Oct 31 03:37:35.402369 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.402362 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696975b8-8ef4-4192-9b87-25f10dc6d138" containerName="manager" Oct 31 03:37:35.402490 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.402465 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="696975b8-8ef4-4192-9b87-25f10dc6d138" containerName="manager" Oct 31 03:37:35.405729 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.405708 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" Oct 31 03:37:35.436379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.436343 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl2c5\" (UniqueName: \"kubernetes.io/projected/c284e318-4c88-4554-bcf3-4ce3da29357a-kube-api-access-hl2c5\") pod \"kserve-controller-manager-776cc66d47-n4sbl\" (UID: \"c284e318-4c88-4554-bcf3-4ce3da29357a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" Oct 31 03:37:35.436528 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.436418 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c284e318-4c88-4554-bcf3-4ce3da29357a-cert\") pod \"kserve-controller-manager-776cc66d47-n4sbl\" (UID: \"c284e318-4c88-4554-bcf3-4ce3da29357a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" Oct 31 03:37:35.501007 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.500962 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 03:37:35.501204 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.501151 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bszz4\"" Oct 31 03:37:35.503560 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.503537 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n4sbl"] Oct 31 03:37:35.510914 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.510890 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj"] Oct 31 03:37:35.514378 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.514356 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:37:35.537031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.536995 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c284e318-4c88-4554-bcf3-4ce3da29357a-cert\") pod \"kserve-controller-manager-776cc66d47-n4sbl\" (UID: \"c284e318-4c88-4554-bcf3-4ce3da29357a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" Oct 31 03:37:35.537221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.537040 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c86e73c-cbbe-445e-8b54-04b127d6fd53-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rcvxj\" (UID: \"5c86e73c-cbbe-445e-8b54-04b127d6fd53\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:37:35.537221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.537070 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2c5\" (UniqueName: \"kubernetes.io/projected/c284e318-4c88-4554-bcf3-4ce3da29357a-kube-api-access-hl2c5\") pod \"kserve-controller-manager-776cc66d47-n4sbl\" (UID: \"c284e318-4c88-4554-bcf3-4ce3da29357a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" Oct 31 03:37:35.537221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.537095 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q87gt\" (UniqueName: \"kubernetes.io/projected/5c86e73c-cbbe-445e-8b54-04b127d6fd53-kube-api-access-q87gt\") pod \"llmisvc-controller-manager-869d5dcc5d-rcvxj\" (UID: \"5c86e73c-cbbe-445e-8b54-04b127d6fd53\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:37:35.539380 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.539355 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c284e318-4c88-4554-bcf3-4ce3da29357a-cert\") pod \"kserve-controller-manager-776cc66d47-n4sbl\" (UID: \"c284e318-4c88-4554-bcf3-4ce3da29357a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" Oct 31 03:37:35.563504 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.563481 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-25d4f\"" Oct 31 03:37:35.563645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.563633 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 03:37:35.570261 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.570236 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj"] Oct 31 03:37:35.596792 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.596757 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2c5\" (UniqueName: \"kubernetes.io/projected/c284e318-4c88-4554-bcf3-4ce3da29357a-kube-api-access-hl2c5\") pod \"kserve-controller-manager-776cc66d47-n4sbl\" (UID: \"c284e318-4c88-4554-bcf3-4ce3da29357a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" Oct 31 03:37:35.619237 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.619196 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-c8lvn"] Oct 31 03:37:35.622929 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.622908 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" Oct 31 03:37:35.638139 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.638094 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c86e73c-cbbe-445e-8b54-04b127d6fd53-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rcvxj\" (UID: \"5c86e73c-cbbe-445e-8b54-04b127d6fd53\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:37:35.638280 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.638159 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjx4q\" (UniqueName: \"kubernetes.io/projected/f6dda95c-c95e-4951-aa12-6809320e9350-kube-api-access-mjx4q\") pod \"model-serving-api-75bd89fdfc-c8lvn\" (UID: \"f6dda95c-c95e-4951-aa12-6809320e9350\") " pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" Oct 31 03:37:35.638280 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.638256 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87gt\" (UniqueName: \"kubernetes.io/projected/5c86e73c-cbbe-445e-8b54-04b127d6fd53-kube-api-access-q87gt\") pod \"llmisvc-controller-manager-869d5dcc5d-rcvxj\" (UID: \"5c86e73c-cbbe-445e-8b54-04b127d6fd53\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:37:35.638351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.638314 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6dda95c-c95e-4951-aa12-6809320e9350-tls-certs\") pod \"model-serving-api-75bd89fdfc-c8lvn\" (UID: \"f6dda95c-c95e-4951-aa12-6809320e9350\") " pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" Oct 31 03:37:35.640490 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.640472 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c86e73c-cbbe-445e-8b54-04b127d6fd53-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rcvxj\" (UID: \"5c86e73c-cbbe-445e-8b54-04b127d6fd53\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:37:35.672110 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.672025 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 03:37:35.672286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.672157 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4kmth\"" Oct 31 03:37:35.678581 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.678550 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-c8lvn"] Oct 31 03:37:35.695886 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.695852 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q87gt\" (UniqueName: \"kubernetes.io/projected/5c86e73c-cbbe-445e-8b54-04b127d6fd53-kube-api-access-q87gt\") pod \"llmisvc-controller-manager-869d5dcc5d-rcvxj\" (UID: \"5c86e73c-cbbe-445e-8b54-04b127d6fd53\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:37:35.718882 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.718854 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" Oct 31 03:37:35.729120 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.729084 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-2lsx8"] Oct 31 03:37:35.733576 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.733554 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:35.738869 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.738832 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6dda95c-c95e-4951-aa12-6809320e9350-tls-certs\") pod \"model-serving-api-75bd89fdfc-c8lvn\" (UID: \"f6dda95c-c95e-4951-aa12-6809320e9350\") " pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" Oct 31 03:37:35.738998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.738871 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k94jg\" (UniqueName: \"kubernetes.io/projected/fa07d775-b0fd-4441-b8ff-b639991d326e-kube-api-access-k94jg\") pod \"odh-model-controller-769875db56-2lsx8\" (UID: \"fa07d775-b0fd-4441-b8ff-b639991d326e\") " pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:35.738998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.738977 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa07d775-b0fd-4441-b8ff-b639991d326e-cert\") pod \"odh-model-controller-769875db56-2lsx8\" (UID: \"fa07d775-b0fd-4441-b8ff-b639991d326e\") " pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:35.739110 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.739012 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjx4q\" (UniqueName: \"kubernetes.io/projected/f6dda95c-c95e-4951-aa12-6809320e9350-kube-api-access-mjx4q\") pod \"model-serving-api-75bd89fdfc-c8lvn\" (UID: \"f6dda95c-c95e-4951-aa12-6809320e9350\") " pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" Oct 31 03:37:35.739110 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.739084 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa07d775-b0fd-4441-b8ff-b639991d326e-metrics-certs\") pod \"odh-model-controller-769875db56-2lsx8\" (UID: \"fa07d775-b0fd-4441-b8ff-b639991d326e\") " pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:35.741544 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.741521 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6dda95c-c95e-4951-aa12-6809320e9350-tls-certs\") pod \"model-serving-api-75bd89fdfc-c8lvn\" (UID: \"f6dda95c-c95e-4951-aa12-6809320e9350\") " pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" Oct 31 03:37:35.794155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.794101 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2lsx8"] Oct 31 03:37:35.794400 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.794381 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 03:37:35.794748 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.794729 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jdb7x\"" Oct 31 03:37:35.794839 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.794805 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 03:37:35.797649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.797622 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjx4q\" (UniqueName: \"kubernetes.io/projected/f6dda95c-c95e-4951-aa12-6809320e9350-kube-api-access-mjx4q\") pod \"model-serving-api-75bd89fdfc-c8lvn\" (UID: \"f6dda95c-c95e-4951-aa12-6809320e9350\") " pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" Oct 31 03:37:35.828897 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.828857 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:37:35.840491 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.840459 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k94jg\" (UniqueName: \"kubernetes.io/projected/fa07d775-b0fd-4441-b8ff-b639991d326e-kube-api-access-k94jg\") pod \"odh-model-controller-769875db56-2lsx8\" (UID: \"fa07d775-b0fd-4441-b8ff-b639991d326e\") " pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:35.840665 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.840515 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa07d775-b0fd-4441-b8ff-b639991d326e-cert\") pod \"odh-model-controller-769875db56-2lsx8\" (UID: \"fa07d775-b0fd-4441-b8ff-b639991d326e\") " pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:35.840665 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.840570 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa07d775-b0fd-4441-b8ff-b639991d326e-metrics-certs\") pod \"odh-model-controller-769875db56-2lsx8\" (UID: \"fa07d775-b0fd-4441-b8ff-b639991d326e\") " pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:35.843272 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.843185 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa07d775-b0fd-4441-b8ff-b639991d326e-metrics-certs\") pod \"odh-model-controller-769875db56-2lsx8\" (UID: \"fa07d775-b0fd-4441-b8ff-b639991d326e\") " pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:35.843272 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.843185 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa07d775-b0fd-4441-b8ff-b639991d326e-cert\") pod \"odh-model-controller-769875db56-2lsx8\" (UID: \"fa07d775-b0fd-4441-b8ff-b639991d326e\") " pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:35.898587 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.898555 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k94jg\" (UniqueName: \"kubernetes.io/projected/fa07d775-b0fd-4441-b8ff-b639991d326e-kube-api-access-k94jg\") pod \"odh-model-controller-769875db56-2lsx8\" (UID: \"fa07d775-b0fd-4441-b8ff-b639991d326e\") " pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:35.936302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:35.936216 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" Oct 31 03:37:36.053386 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:36.053351 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:36.090780 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:36.090750 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n4sbl"] Oct 31 03:37:36.198423 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:36.198399 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj"] Oct 31 03:37:36.199981 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:37:36.199951 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c86e73c_cbbe_445e_8b54_04b127d6fd53.slice/crio-eb1cb2886cc16477d266714810050198ff5369ca78da15564ee9694ce6298416 WatchSource:0}: Error finding container eb1cb2886cc16477d266714810050198ff5369ca78da15564ee9694ce6298416: Status 404 returned error can't find the container with id eb1cb2886cc16477d266714810050198ff5369ca78da15564ee9694ce6298416 Oct 31 03:37:36.303120 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:36.303094 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-c8lvn"] Oct 31 03:37:36.304714 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:37:36.304683 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dda95c_c95e_4951_aa12_6809320e9350.slice/crio-484a9f6fb4fc889a2b60b032ab82f9e4ab883ada7269f4229c92dbef3a39d89b WatchSource:0}: Error finding container 484a9f6fb4fc889a2b60b032ab82f9e4ab883ada7269f4229c92dbef3a39d89b: Status 404 returned error can't find the container with id 484a9f6fb4fc889a2b60b032ab82f9e4ab883ada7269f4229c92dbef3a39d89b Oct 31 03:37:36.422076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:36.422047 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2lsx8"] Oct 31 03:37:36.422656 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:37:36.422630 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa07d775_b0fd_4441_b8ff_b639991d326e.slice/crio-f57cd8de75a1d7bdbe0d8756f2c9b94af5f7751c14d0a1cec597c9997a9b803b WatchSource:0}: Error finding container f57cd8de75a1d7bdbe0d8756f2c9b94af5f7751c14d0a1cec597c9997a9b803b: Status 404 returned error can't find the container with id f57cd8de75a1d7bdbe0d8756f2c9b94af5f7751c14d0a1cec597c9997a9b803b Oct 31 03:37:37.041855 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:37.041816 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" event={"ID":"5c86e73c-cbbe-445e-8b54-04b127d6fd53","Type":"ContainerStarted","Data":"eb1cb2886cc16477d266714810050198ff5369ca78da15564ee9694ce6298416"} Oct 31 03:37:37.043097 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:37.043065 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2lsx8" event={"ID":"fa07d775-b0fd-4441-b8ff-b639991d326e","Type":"ContainerStarted","Data":"f57cd8de75a1d7bdbe0d8756f2c9b94af5f7751c14d0a1cec597c9997a9b803b"} Oct 31 03:37:37.044361 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:37.044337 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" event={"ID":"f6dda95c-c95e-4951-aa12-6809320e9350","Type":"ContainerStarted","Data":"484a9f6fb4fc889a2b60b032ab82f9e4ab883ada7269f4229c92dbef3a39d89b"} Oct 31 03:37:37.045448 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:37.045427 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" event={"ID":"c284e318-4c88-4554-bcf3-4ce3da29357a","Type":"ContainerStarted","Data":"f2b1ac38b6d07200471fca22357f333e5d2a439587d94a0aedcec1e30010a55a"} Oct 31 03:37:40.071734 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:40.071696 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" event={"ID":"f6dda95c-c95e-4951-aa12-6809320e9350","Type":"ContainerStarted","Data":"786ec00479b17ff79215a63403731743cb713331823798c606cc6ed2316f51af"} Oct 31 03:37:40.072271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:40.071848 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" Oct 31 03:37:40.186825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:40.186704 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" podStartSLOduration=1.558691107 podStartE2EDuration="5.186679318s" totalImagesPullingTime="3.627988211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:40.182926168 +0000 UTC m=+557.183551397" watchObservedRunningTime="2026-10-31 03:37:40.186679318 +0000 UTC m=+557.187304548" Oct 31 03:37:41.015158 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:41.014454 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-l7zq6" Oct 31 03:37:41.081114 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:41.081081 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" event={"ID":"c284e318-4c88-4554-bcf3-4ce3da29357a","Type":"ContainerStarted","Data":"e4b17d1a1622576b6716e24218320b863869a34a76f5f83ef43d4aad940191ae"} Oct 31 03:37:41.236345 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:41.236282 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" podStartSLOduration=1.3747710180000001 podStartE2EDuration="6.236261016s" totalImagesPullingTime="4.861489998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:41.233293307 +0000 UTC m=+558.233918537" watchObservedRunningTime="2026-10-31 03:37:41.236261016 +0000 UTC m=+558.236886245" Oct 31 03:37:42.088184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:42.088145 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" event={"ID":"5c86e73c-cbbe-445e-8b54-04b127d6fd53","Type":"ContainerStarted","Data":"bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923"} Oct 31 03:37:42.088787 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:42.088310 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:37:42.089773 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:42.089743 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2lsx8" event={"ID":"fa07d775-b0fd-4441-b8ff-b639991d326e","Type":"ContainerStarted","Data":"e8b2181b972b47d94f977219234b8a7385068364418944f18d28cf038bab9766"} Oct 31 03:37:42.089901 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:42.089798 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:37:42.089901 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:42.089882 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" Oct 31 03:37:42.197560 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:42.197495 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" podStartSLOduration=2.217423655 podStartE2EDuration="7.197479483s" totalImagesPullingTime="4.980055828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:42.196944713 +0000 UTC m=+559.197569943" watchObservedRunningTime="2026-10-31 03:37:42.197479483 +0000 UTC m=+559.198104712" Oct 31 03:37:42.312576 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:42.312512 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-2lsx8" podStartSLOduration=2.542580636 podStartE2EDuration="7.312492998s" totalImagesPullingTime="4.769912362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:42.310726984 +0000 UTC m=+559.311352212" watchObservedRunningTime="2026-10-31 03:37:42.312492998 +0000 UTC m=+559.313118226" Oct 31 03:37:49.938212 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:49.938173 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:37:51.085373 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:51.085344 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-c8lvn" Oct 31 03:37:53.097134 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:37:53.097102 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-2lsx8" Oct 31 03:38:00.202305 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:00.202267 4426 generic.go:356] "Generic (PLEG): container finished" podID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerID="7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b" exitCode=0 Oct 31 03:38:00.202737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:00.202340 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" event={"ID":"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea","Type":"ContainerDied","Data":"7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b"} Oct 31 03:38:00.202737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:00.202701 4426 scope.go:122] "RemoveContainer" containerID="7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b" Oct 31 03:38:00.203880 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:00.203847 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:38:00.203937 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:00.203923 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:38:00.203981 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:00.203958 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:38:00.760075 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:00.760035 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:38:01.209687 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:01.209650 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" event={"ID":"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea","Type":"ContainerStarted","Data":"a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f"} Oct 31 03:38:02.136153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:02.124517 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:38:02.139942 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:02.139902 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-kjrmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 31 03:38:02.140265 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:02.140225 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 31 03:38:02.863997 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:38:02.863962 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:38:03.139544 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:03.139510 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-kjrmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 31 03:38:03.139737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:03.139579 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 31 03:38:04.136150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:04.136055 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-kjrmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 31 03:38:04.136150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:04.136148 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 31 03:38:05.133151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:05.131546 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-kjrmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 31 03:38:05.133151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:05.131612 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 31 03:38:06.131699 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:06.131608 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-kjrmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 31 03:38:06.132419 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:06.132386 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 31 03:38:07.128366 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.128338 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:38:07.128622 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.128601 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:38:07.129279 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.129259 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:38:07.373706 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.373672 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-dspts"] Oct 31 03:38:07.385484 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.385417 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-dspts" Oct 31 03:38:07.432018 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.431983 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-dspts"] Oct 31 03:38:07.434606 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.434580 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 03:38:07.434760 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.434628 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ppjlh\"" Oct 31 03:38:07.584363 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.584327 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e93e963-9202-4fd0-9c36-6227d999bc8b-tls-cert\") pod \"authorino-6dd76875b5-dspts\" (UID: \"9e93e963-9202-4fd0-9c36-6227d999bc8b\") " pod="kuadrant-system/authorino-6dd76875b5-dspts" Oct 31 03:38:07.584551 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.584449 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsfxq\" (UniqueName: \"kubernetes.io/projected/9e93e963-9202-4fd0-9c36-6227d999bc8b-kube-api-access-rsfxq\") pod \"authorino-6dd76875b5-dspts\" (UID: \"9e93e963-9202-4fd0-9c36-6227d999bc8b\") " pod="kuadrant-system/authorino-6dd76875b5-dspts" Oct 31 03:38:07.685657 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.685617 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsfxq\" (UniqueName: \"kubernetes.io/projected/9e93e963-9202-4fd0-9c36-6227d999bc8b-kube-api-access-rsfxq\") pod \"authorino-6dd76875b5-dspts\" (UID: \"9e93e963-9202-4fd0-9c36-6227d999bc8b\") " pod="kuadrant-system/authorino-6dd76875b5-dspts" Oct 31 03:38:07.685852 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.685781 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e93e963-9202-4fd0-9c36-6227d999bc8b-tls-cert\") pod \"authorino-6dd76875b5-dspts\" (UID: \"9e93e963-9202-4fd0-9c36-6227d999bc8b\") " pod="kuadrant-system/authorino-6dd76875b5-dspts" Oct 31 03:38:07.688278 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.688254 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e93e963-9202-4fd0-9c36-6227d999bc8b-tls-cert\") pod \"authorino-6dd76875b5-dspts\" (UID: \"9e93e963-9202-4fd0-9c36-6227d999bc8b\") " pod="kuadrant-system/authorino-6dd76875b5-dspts" Oct 31 03:38:07.743920 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.743887 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsfxq\" (UniqueName: \"kubernetes.io/projected/9e93e963-9202-4fd0-9c36-6227d999bc8b-kube-api-access-rsfxq\") pod \"authorino-6dd76875b5-dspts\" (UID: \"9e93e963-9202-4fd0-9c36-6227d999bc8b\") " pod="kuadrant-system/authorino-6dd76875b5-dspts" Oct 31 03:38:07.997457 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:07.997369 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-dspts" Oct 31 03:38:08.361583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:08.361559 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-dspts"] Oct 31 03:38:08.363442 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:38:08.363408 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e93e963_9202_4fd0_9c36_6227d999bc8b.slice/crio-9cb2a931abdf89998b9c9888d2bd1d8cb9f82535ca05afd79987ab46db880b07 WatchSource:0}: Error finding container 9cb2a931abdf89998b9c9888d2bd1d8cb9f82535ca05afd79987ab46db880b07: Status 404 returned error can't find the container with id 9cb2a931abdf89998b9c9888d2bd1d8cb9f82535ca05afd79987ab46db880b07 Oct 31 03:38:09.277676 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:09.277637 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd76875b5-dspts" event={"ID":"9e93e963-9202-4fd0-9c36-6227d999bc8b","Type":"ContainerStarted","Data":"9cb2a931abdf89998b9c9888d2bd1d8cb9f82535ca05afd79987ab46db880b07"} Oct 31 03:38:10.285221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:10.285135 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd76875b5-dspts" event={"ID":"9e93e963-9202-4fd0-9c36-6227d999bc8b","Type":"ContainerStarted","Data":"c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e"} Oct 31 03:38:10.394056 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:10.394002 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6dd76875b5-dspts" podStartSLOduration=1.737761273 podStartE2EDuration="3.393987349s" totalImagesPullingTime="1.656226076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:10.392091273 +0000 UTC m=+587.392716500" watchObservedRunningTime="2026-10-31 03:38:10.393987349 +0000 UTC m=+587.394612577" Oct 31 03:38:13.097019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:13.096987 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:38:13.099849 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:13.099830 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-n4sbl" Oct 31 03:38:16.586735 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.586678 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bd2sw"] Oct 31 03:38:16.594419 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.594390 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" Oct 31 03:38:16.645072 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.645033 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bd2sw"] Oct 31 03:38:16.667838 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.667796 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5656c888-d263-4107-8c9d-933f036dd4c0-tls-cert\") pod \"authorino-6889cbd8c4-bd2sw\" (UID: \"5656c888-d263-4107-8c9d-933f036dd4c0\") " pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" Oct 31 03:38:16.668013 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.667953 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m44gv\" (UniqueName: \"kubernetes.io/projected/5656c888-d263-4107-8c9d-933f036dd4c0-kube-api-access-m44gv\") pod \"authorino-6889cbd8c4-bd2sw\" (UID: \"5656c888-d263-4107-8c9d-933f036dd4c0\") " pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" Oct 31 03:38:16.768891 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.768852 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5656c888-d263-4107-8c9d-933f036dd4c0-tls-cert\") pod \"authorino-6889cbd8c4-bd2sw\" (UID: \"5656c888-d263-4107-8c9d-933f036dd4c0\") " pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" Oct 31 03:38:16.769053 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.768957 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m44gv\" (UniqueName: \"kubernetes.io/projected/5656c888-d263-4107-8c9d-933f036dd4c0-kube-api-access-m44gv\") pod \"authorino-6889cbd8c4-bd2sw\" (UID: \"5656c888-d263-4107-8c9d-933f036dd4c0\") " pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" Oct 31 03:38:16.771274 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.771250 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5656c888-d263-4107-8c9d-933f036dd4c0-tls-cert\") pod \"authorino-6889cbd8c4-bd2sw\" (UID: \"5656c888-d263-4107-8c9d-933f036dd4c0\") " pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" Oct 31 03:38:16.826602 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.826564 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m44gv\" (UniqueName: \"kubernetes.io/projected/5656c888-d263-4107-8c9d-933f036dd4c0-kube-api-access-m44gv\") pod \"authorino-6889cbd8c4-bd2sw\" (UID: \"5656c888-d263-4107-8c9d-933f036dd4c0\") " pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" Oct 31 03:38:16.906456 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.906415 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" Oct 31 03:38:16.973181 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.970877 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bd2sw"] Oct 31 03:38:16.998855 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:16.998825 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78db76d6d7-vt4tz"] Oct 31 03:38:17.003066 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.003050 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78db76d6d7-vt4tz" Oct 31 03:38:17.056809 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.056779 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78db76d6d7-vt4tz"] Oct 31 03:38:17.072550 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.072516 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nklh4\" (UniqueName: \"kubernetes.io/projected/883eeb42-2686-4880-87a0-650aa653e124-kube-api-access-nklh4\") pod \"authorino-78db76d6d7-vt4tz\" (UID: \"883eeb42-2686-4880-87a0-650aa653e124\") " pod="kuadrant-system/authorino-78db76d6d7-vt4tz" Oct 31 03:38:17.072714 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.072592 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/883eeb42-2686-4880-87a0-650aa653e124-tls-cert\") pod \"authorino-78db76d6d7-vt4tz\" (UID: \"883eeb42-2686-4880-87a0-650aa653e124\") " pod="kuadrant-system/authorino-78db76d6d7-vt4tz" Oct 31 03:38:17.173419 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.173327 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/883eeb42-2686-4880-87a0-650aa653e124-tls-cert\") pod \"authorino-78db76d6d7-vt4tz\" (UID: \"883eeb42-2686-4880-87a0-650aa653e124\") " pod="kuadrant-system/authorino-78db76d6d7-vt4tz" Oct 31 03:38:17.173599 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.173498 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nklh4\" (UniqueName: \"kubernetes.io/projected/883eeb42-2686-4880-87a0-650aa653e124-kube-api-access-nklh4\") pod \"authorino-78db76d6d7-vt4tz\" (UID: \"883eeb42-2686-4880-87a0-650aa653e124\") " pod="kuadrant-system/authorino-78db76d6d7-vt4tz" Oct 31 03:38:17.175829 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.175801 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/883eeb42-2686-4880-87a0-650aa653e124-tls-cert\") pod \"authorino-78db76d6d7-vt4tz\" (UID: \"883eeb42-2686-4880-87a0-650aa653e124\") " pod="kuadrant-system/authorino-78db76d6d7-vt4tz" Oct 31 03:38:17.230874 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.230843 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nklh4\" (UniqueName: \"kubernetes.io/projected/883eeb42-2686-4880-87a0-650aa653e124-kube-api-access-nklh4\") pod \"authorino-78db76d6d7-vt4tz\" (UID: \"883eeb42-2686-4880-87a0-650aa653e124\") " pod="kuadrant-system/authorino-78db76d6d7-vt4tz" Oct 31 03:38:17.270653 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.270629 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bd2sw"] Oct 31 03:38:17.272149 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:38:17.272106 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5656c888_d263_4107_8c9d_933f036dd4c0.slice/crio-4c258a9c18d3a7746dbdd4a97dffef8f405b19136e0a4fd5a3cf2a19c33329fc WatchSource:0}: Error finding container 4c258a9c18d3a7746dbdd4a97dffef8f405b19136e0a4fd5a3cf2a19c33329fc: Status 404 returned error can't find the container with id 4c258a9c18d3a7746dbdd4a97dffef8f405b19136e0a4fd5a3cf2a19c33329fc Oct 31 03:38:17.315231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.315198 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78db76d6d7-vt4tz" Oct 31 03:38:17.322891 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.322854 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" event={"ID":"5656c888-d263-4107-8c9d-933f036dd4c0","Type":"ContainerStarted","Data":"4c258a9c18d3a7746dbdd4a97dffef8f405b19136e0a4fd5a3cf2a19c33329fc"} Oct 31 03:38:17.679512 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:17.679483 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78db76d6d7-vt4tz"] Oct 31 03:38:17.681027 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:38:17.681000 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod883eeb42_2686_4880_87a0_650aa653e124.slice/crio-e99db5d71351f49d48655048d3ae40e98d2f60de7155e80b0c40ba8c5ec0decd WatchSource:0}: Error finding container e99db5d71351f49d48655048d3ae40e98d2f60de7155e80b0c40ba8c5ec0decd: Status 404 returned error can't find the container with id e99db5d71351f49d48655048d3ae40e98d2f60de7155e80b0c40ba8c5ec0decd Oct 31 03:38:18.330227 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:18.330184 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78db76d6d7-vt4tz" event={"ID":"883eeb42-2686-4880-87a0-650aa653e124","Type":"ContainerStarted","Data":"e99db5d71351f49d48655048d3ae40e98d2f60de7155e80b0c40ba8c5ec0decd"} Oct 31 03:38:19.337724 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.337685 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78db76d6d7-vt4tz" event={"ID":"883eeb42-2686-4880-87a0-650aa653e124","Type":"ContainerStarted","Data":"2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52"} Oct 31 03:38:19.339394 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.339357 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" event={"ID":"5656c888-d263-4107-8c9d-933f036dd4c0","Type":"ContainerStarted","Data":"88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8"} Oct 31 03:38:19.339524 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.339391 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" podUID="5656c888-d263-4107-8c9d-933f036dd4c0" containerName="authorino" containerID="cri-o://88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8" gracePeriod=30 Oct 31 03:38:19.448584 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.448526 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-78db76d6d7-vt4tz" podStartSLOduration=2.241529431 podStartE2EDuration="3.448509466s" totalImagesPullingTime="1.206980035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:19.445187016 +0000 UTC m=+596.445812247" watchObservedRunningTime="2026-10-31 03:38:19.448509466 +0000 UTC m=+596.449134714" Oct 31 03:38:19.488577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.482251 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-dspts"] Oct 31 03:38:19.488577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.482489 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6dd76875b5-dspts" podUID="9e93e963-9202-4fd0-9c36-6227d999bc8b" containerName="authorino" containerID="cri-o://c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e" gracePeriod=30 Oct 31 03:38:19.556597 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.556530 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" podStartSLOduration=2.119361599 podStartE2EDuration="3.556507541s" totalImagesPullingTime="1.437145942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:19.553770395 +0000 UTC m=+596.554395623" watchObservedRunningTime="2026-10-31 03:38:19.556507541 +0000 UTC m=+596.557132772" Oct 31 03:38:19.621444 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.621407 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" Oct 31 03:38:19.704415 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.704382 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5656c888-d263-4107-8c9d-933f036dd4c0-tls-cert\" (UniqueName: \"kubernetes.io/secret/5656c888-d263-4107-8c9d-933f036dd4c0-tls-cert\") pod \"5656c888-d263-4107-8c9d-933f036dd4c0\" (UID: \"5656c888-d263-4107-8c9d-933f036dd4c0\") " Oct 31 03:38:19.704602 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.704481 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5656c888-d263-4107-8c9d-933f036dd4c0-kube-api-access-m44gv\" (UniqueName: \"kubernetes.io/projected/5656c888-d263-4107-8c9d-933f036dd4c0-kube-api-access-m44gv\") pod \"5656c888-d263-4107-8c9d-933f036dd4c0\" (UID: \"5656c888-d263-4107-8c9d-933f036dd4c0\") " Oct 31 03:38:19.719624 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.719571 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5656c888-d263-4107-8c9d-933f036dd4c0-kube-api-access-m44gv" pod "5656c888-d263-4107-8c9d-933f036dd4c0" (UID: "5656c888-d263-4107-8c9d-933f036dd4c0"). InnerVolumeSpecName "kube-api-access-m44gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:19.723285 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.723252 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5656c888-d263-4107-8c9d-933f036dd4c0-tls-cert" pod "5656c888-d263-4107-8c9d-933f036dd4c0" (UID: "5656c888-d263-4107-8c9d-933f036dd4c0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:19.741280 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.741255 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-dspts" Oct 31 03:38:19.806366 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.806328 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e93e963-9202-4fd0-9c36-6227d999bc8b-kube-api-access-rsfxq\" (UniqueName: \"kubernetes.io/projected/9e93e963-9202-4fd0-9c36-6227d999bc8b-kube-api-access-rsfxq\") pod \"9e93e963-9202-4fd0-9c36-6227d999bc8b\" (UID: \"9e93e963-9202-4fd0-9c36-6227d999bc8b\") " Oct 31 03:38:19.806535 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.806420 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e93e963-9202-4fd0-9c36-6227d999bc8b-tls-cert\" (UniqueName: \"kubernetes.io/secret/9e93e963-9202-4fd0-9c36-6227d999bc8b-tls-cert\") pod \"9e93e963-9202-4fd0-9c36-6227d999bc8b\" (UID: \"9e93e963-9202-4fd0-9c36-6227d999bc8b\") " Oct 31 03:38:19.806687 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.806668 4426 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5656c888-d263-4107-8c9d-933f036dd4c0-tls-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:19.806734 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.806697 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m44gv\" (UniqueName: \"kubernetes.io/projected/5656c888-d263-4107-8c9d-933f036dd4c0-kube-api-access-m44gv\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:19.808580 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.808541 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e93e963-9202-4fd0-9c36-6227d999bc8b-kube-api-access-rsfxq" pod "9e93e963-9202-4fd0-9c36-6227d999bc8b" (UID: "9e93e963-9202-4fd0-9c36-6227d999bc8b"). InnerVolumeSpecName "kube-api-access-rsfxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:19.816757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.816721 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e93e963-9202-4fd0-9c36-6227d999bc8b-tls-cert" pod "9e93e963-9202-4fd0-9c36-6227d999bc8b" (UID: "9e93e963-9202-4fd0-9c36-6227d999bc8b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:19.907625 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.907592 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsfxq\" (UniqueName: \"kubernetes.io/projected/9e93e963-9202-4fd0-9c36-6227d999bc8b-kube-api-access-rsfxq\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:19.907625 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:19.907626 4426 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e93e963-9202-4fd0-9c36-6227d999bc8b-tls-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:20.346313 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.346230 4426 generic.go:356] "Generic (PLEG): container finished" podID="9e93e963-9202-4fd0-9c36-6227d999bc8b" containerID="c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e" exitCode=0 Oct 31 03:38:20.346313 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.346278 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-dspts" Oct 31 03:38:20.346791 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.346309 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd76875b5-dspts" event={"ID":"9e93e963-9202-4fd0-9c36-6227d999bc8b","Type":"ContainerDied","Data":"c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e"} Oct 31 03:38:20.346791 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.346347 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd76875b5-dspts" event={"ID":"9e93e963-9202-4fd0-9c36-6227d999bc8b","Type":"ContainerDied","Data":"9cb2a931abdf89998b9c9888d2bd1d8cb9f82535ca05afd79987ab46db880b07"} Oct 31 03:38:20.346791 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.346364 4426 scope.go:122] "RemoveContainer" containerID="c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e" Oct 31 03:38:20.347717 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.347695 4426 generic.go:356] "Generic (PLEG): container finished" podID="5656c888-d263-4107-8c9d-933f036dd4c0" containerID="88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8" exitCode=0 Oct 31 03:38:20.347812 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.347742 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" Oct 31 03:38:20.347812 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.347781 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" event={"ID":"5656c888-d263-4107-8c9d-933f036dd4c0","Type":"ContainerDied","Data":"88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8"} Oct 31 03:38:20.347812 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.347804 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bd2sw" event={"ID":"5656c888-d263-4107-8c9d-933f036dd4c0","Type":"ContainerDied","Data":"4c258a9c18d3a7746dbdd4a97dffef8f405b19136e0a4fd5a3cf2a19c33329fc"} Oct 31 03:38:20.355225 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.355203 4426 scope.go:122] "RemoveContainer" containerID="c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e" Oct 31 03:38:20.355501 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:38:20.355482 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e\": container with ID starting with c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e not found: ID does not exist" containerID="c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e" Oct 31 03:38:20.355577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.355514 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e"} err="failed to get container status \"c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e\": rpc error: code = NotFound desc = could not find container \"c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e\": container with ID starting with c499b7b3847ebb6d3f35b735ed6cfca3d516e4264351d1538dd6db178ffbcb3e not found: ID does not exist" Oct 31 03:38:20.355577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.355538 4426 scope.go:122] "RemoveContainer" containerID="88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8" Oct 31 03:38:20.364146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.364103 4426 scope.go:122] "RemoveContainer" containerID="88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8" Oct 31 03:38:20.364905 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:38:20.364886 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8\": container with ID starting with 88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8 not found: ID does not exist" containerID="88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8" Oct 31 03:38:20.364979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.364913 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8"} err="failed to get container status \"88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8\": rpc error: code = NotFound desc = could not find container \"88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8\": container with ID starting with 88ca25d9f1d252464f38b8dc21a1826a10ef04122eed43008eb58fec7e02dfb8 not found: ID does not exist" Oct 31 03:38:20.500143 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.500098 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bd2sw"] Oct 31 03:38:20.504002 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.503977 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bd2sw"] Oct 31 03:38:20.607073 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.606986 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-dspts"] Oct 31 03:38:20.610728 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:20.610701 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-dspts"] Oct 31 03:38:21.128043 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:21.128011 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5656c888-d263-4107-8c9d-933f036dd4c0" path="/var/lib/kubelet/pods/5656c888-d263-4107-8c9d-933f036dd4c0/volumes" Oct 31 03:38:21.128409 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:21.128394 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e93e963-9202-4fd0-9c36-6227d999bc8b" path="/var/lib/kubelet/pods/9e93e963-9202-4fd0-9c36-6227d999bc8b/volumes" Oct 31 03:38:23.158706 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:23.158676 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:38:23.159287 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:38:23.159270 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:38:32.846619 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:38:32.846582 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:39:02.844043 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:39:02.844001 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:39:15.067880 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.067845 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj"] Oct 31 03:39:15.068487 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.068066 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" podUID="5c86e73c-cbbe-445e-8b54-04b127d6fd53" containerName="manager" containerID="cri-o://bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923" gracePeriod=30 Oct 31 03:39:15.316564 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.316538 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:39:15.407092 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.407054 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c86e73c-cbbe-445e-8b54-04b127d6fd53-cert\" (UniqueName: \"kubernetes.io/secret/5c86e73c-cbbe-445e-8b54-04b127d6fd53-cert\") pod \"5c86e73c-cbbe-445e-8b54-04b127d6fd53\" (UID: \"5c86e73c-cbbe-445e-8b54-04b127d6fd53\") " Oct 31 03:39:15.407277 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.407156 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c86e73c-cbbe-445e-8b54-04b127d6fd53-kube-api-access-q87gt\" (UniqueName: \"kubernetes.io/projected/5c86e73c-cbbe-445e-8b54-04b127d6fd53-kube-api-access-q87gt\") pod \"5c86e73c-cbbe-445e-8b54-04b127d6fd53\" (UID: \"5c86e73c-cbbe-445e-8b54-04b127d6fd53\") " Oct 31 03:39:15.409050 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.409019 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c86e73c-cbbe-445e-8b54-04b127d6fd53-cert" pod "5c86e73c-cbbe-445e-8b54-04b127d6fd53" (UID: "5c86e73c-cbbe-445e-8b54-04b127d6fd53"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:39:15.409180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.409159 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c86e73c-cbbe-445e-8b54-04b127d6fd53-kube-api-access-q87gt" pod "5c86e73c-cbbe-445e-8b54-04b127d6fd53" (UID: "5c86e73c-cbbe-445e-8b54-04b127d6fd53"). InnerVolumeSpecName "kube-api-access-q87gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:15.507984 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.507946 4426 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c86e73c-cbbe-445e-8b54-04b127d6fd53-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:15.507984 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.507978 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q87gt\" (UniqueName: \"kubernetes.io/projected/5c86e73c-cbbe-445e-8b54-04b127d6fd53-kube-api-access-q87gt\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:15.642691 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.642658 4426 generic.go:356] "Generic (PLEG): container finished" podID="5c86e73c-cbbe-445e-8b54-04b127d6fd53" containerID="bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923" exitCode=0 Oct 31 03:39:15.642839 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.642701 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" Oct 31 03:39:15.642839 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.642737 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" event={"ID":"5c86e73c-cbbe-445e-8b54-04b127d6fd53","Type":"ContainerDied","Data":"bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923"} Oct 31 03:39:15.642839 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.642794 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj" event={"ID":"5c86e73c-cbbe-445e-8b54-04b127d6fd53","Type":"ContainerDied","Data":"eb1cb2886cc16477d266714810050198ff5369ca78da15564ee9694ce6298416"} Oct 31 03:39:15.642839 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.642817 4426 scope.go:122] "RemoveContainer" containerID="bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923" Oct 31 03:39:15.652293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.652274 4426 scope.go:122] "RemoveContainer" containerID="bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923" Oct 31 03:39:15.652530 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:39:15.652511 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923\": container with ID starting with bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923 not found: ID does not exist" containerID="bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923" Oct 31 03:39:15.652581 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.652539 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923"} err="failed to get container status \"bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923\": rpc error: code = NotFound desc = could not find container \"bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923\": container with ID starting with bcd1135d37d051e08d283b6bb5bb5aab31e6e02fb2639c854f1bfea2f553c923 not found: ID does not exist" Oct 31 03:39:15.797944 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.797852 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj"] Oct 31 03:39:15.844099 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:15.844059 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rcvxj"] Oct 31 03:39:17.129211 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:17.129180 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c86e73c-cbbe-445e-8b54-04b127d6fd53" path="/var/lib/kubelet/pods/5c86e73c-cbbe-445e-8b54-04b127d6fd53/volumes" Oct 31 03:39:32.845489 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:39:32.845451 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:39:37.726362 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.726279 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b"] Oct 31 03:39:37.726757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.726691 4426 cpu_manager.go:414] "containerMap: removing container" podUID="9e93e963-9202-4fd0-9c36-6227d999bc8b" containerName="authorino" Oct 31 03:39:37.726757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.726704 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e93e963-9202-4fd0-9c36-6227d999bc8b" containerName="authorino" Oct 31 03:39:37.726757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.726714 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5656c888-d263-4107-8c9d-933f036dd4c0" containerName="authorino" Oct 31 03:39:37.726757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.726719 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5656c888-d263-4107-8c9d-933f036dd4c0" containerName="authorino" Oct 31 03:39:37.726757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.726729 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5c86e73c-cbbe-445e-8b54-04b127d6fd53" containerName="manager" Oct 31 03:39:37.726757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.726733 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c86e73c-cbbe-445e-8b54-04b127d6fd53" containerName="manager" Oct 31 03:39:37.726945 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.726810 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e93e963-9202-4fd0-9c36-6227d999bc8b" containerName="authorino" Oct 31 03:39:37.726945 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.726817 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="5656c888-d263-4107-8c9d-933f036dd4c0" containerName="authorino" Oct 31 03:39:37.726945 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.726823 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c86e73c-cbbe-445e-8b54-04b127d6fd53" containerName="manager" Oct 31 03:39:37.731279 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.731260 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:37.780412 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.780372 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 03:39:37.780581 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.780453 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 03:39:37.827009 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.826977 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5hbl9\"" Oct 31 03:39:37.827228 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.827066 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 31 03:39:37.829324 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.829300 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b"] Oct 31 03:39:37.909610 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.909573 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:37.909610 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.909612 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b3bae1-fa55-4a39-873b-69c1da86ecf5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:37.909845 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.909646 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:37.909845 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.909733 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:37.909845 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.909776 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85vzp\" (UniqueName: \"kubernetes.io/projected/71b3bae1-fa55-4a39-873b-69c1da86ecf5-kube-api-access-85vzp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:37.909845 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:37.909828 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.011044 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.010949 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.011044 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.010997 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.011044 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.011016 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b3bae1-fa55-4a39-873b-69c1da86ecf5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.011044 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.011045 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.011425 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.011076 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.011425 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.011108 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85vzp\" (UniqueName: \"kubernetes.io/projected/71b3bae1-fa55-4a39-873b-69c1da86ecf5-kube-api-access-85vzp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.011536 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.011420 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.011536 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.011476 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.011603 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.011534 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.013398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.013371 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b3bae1-fa55-4a39-873b-69c1da86ecf5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.013691 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.013668 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b3bae1-fa55-4a39-873b-69c1da86ecf5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.071388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.071355 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85vzp\" (UniqueName: \"kubernetes.io/projected/71b3bae1-fa55-4a39-873b-69c1da86ecf5-kube-api-access-85vzp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bc65b\" (UID: \"71b3bae1-fa55-4a39-873b-69c1da86ecf5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.344295 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.344208 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:38.722799 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.722775 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b"] Oct 31 03:39:38.724911 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:39:38.724860 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b3bae1_fa55_4a39_873b_69c1da86ecf5.slice/crio-c61c56cf91dbe040ee84059bcef3df9d6df124f40b70c6c6969bb3d7e4de1f01 WatchSource:0}: Error finding container c61c56cf91dbe040ee84059bcef3df9d6df124f40b70c6c6969bb3d7e4de1f01: Status 404 returned error can't find the container with id c61c56cf91dbe040ee84059bcef3df9d6df124f40b70c6c6969bb3d7e4de1f01 Oct 31 03:39:38.726225 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.726209 4426 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:39:38.763911 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:38.763877 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" event={"ID":"71b3bae1-fa55-4a39-873b-69c1da86ecf5","Type":"ContainerStarted","Data":"c61c56cf91dbe040ee84059bcef3df9d6df124f40b70c6c6969bb3d7e4de1f01"} Oct 31 03:39:40.784613 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:40.784573 4426 generic.go:356] "Generic (PLEG): container finished" podID="2df4691e-00da-49e4-bb70-02ad5fad1625" containerID="22b50971fa3d01f6d36fd0d59a0d1e422c1cedaba6f9a9b4998c815328c8e5fc" exitCode=1 Oct 31 03:39:40.785095 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:40.784662 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-565x2" event={"ID":"2df4691e-00da-49e4-bb70-02ad5fad1625","Type":"ContainerDied","Data":"22b50971fa3d01f6d36fd0d59a0d1e422c1cedaba6f9a9b4998c815328c8e5fc"} Oct 31 03:39:40.785203 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:40.785151 4426 scope.go:122] "RemoveContainer" containerID="22b50971fa3d01f6d36fd0d59a0d1e422c1cedaba6f9a9b4998c815328c8e5fc" Oct 31 03:39:41.079393 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.079358 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:39:41.616853 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.616815 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qlm9m"] Oct 31 03:39:41.620725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.620697 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.649500 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.649456 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.649660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.649547 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.649737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.649656 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czmw\" (UniqueName: \"kubernetes.io/projected/33c358fc-a3bb-4db1-80b3-d1f799e29621-kube-api-access-4czmw\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.649737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.649694 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-home\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.649737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.649716 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c358fc-a3bb-4db1-80b3-d1f799e29621-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.649918 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.649893 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.670225 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.670192 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 03:39:41.676800 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.676771 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qlm9m"] Oct 31 03:39:41.750311 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.750272 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.750510 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.750323 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.750510 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.750360 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czmw\" (UniqueName: \"kubernetes.io/projected/33c358fc-a3bb-4db1-80b3-d1f799e29621-kube-api-access-4czmw\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.750510 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.750379 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-home\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.750510 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.750399 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c358fc-a3bb-4db1-80b3-d1f799e29621-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.750510 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.750439 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.750848 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.750825 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.750926 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.750874 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.750926 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.750825 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-home\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.752687 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.752662 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c358fc-a3bb-4db1-80b3-d1f799e29621-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.752811 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.752749 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c358fc-a3bb-4db1-80b3-d1f799e29621-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.791853 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.791817 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" event={"ID":"71b3bae1-fa55-4a39-873b-69c1da86ecf5","Type":"ContainerStarted","Data":"0bea5357161260db5807ff3415889dc60d165910de3c10032444672414f2e73c"} Oct 31 03:39:41.808235 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.808189 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czmw\" (UniqueName: \"kubernetes.io/projected/33c358fc-a3bb-4db1-80b3-d1f799e29621-kube-api-access-4czmw\") pod \"echo-simulator-kserve-64b6f8f96c-qlm9m\" (UID: \"33c358fc-a3bb-4db1-80b3-d1f799e29621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:41.903661 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.903607 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" podStartSLOduration=2.65392569 podStartE2EDuration="4.90359033s" totalImagesPullingTime="2.24966464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:41.902182883 +0000 UTC m=+678.902808112" watchObservedRunningTime="2026-10-31 03:39:41.90359033 +0000 UTC m=+678.904215558" Oct 31 03:39:41.934572 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:41.934542 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:42.303299 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:42.303267 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qlm9m"] Oct 31 03:39:42.304952 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:39:42.304926 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c358fc_a3bb_4db1_80b3_d1f799e29621.slice/crio-a4a09fc53e7eef13139d8542ec9e4a7824b28b0136bd5d02ba6698f59f8398cb WatchSource:0}: Error finding container a4a09fc53e7eef13139d8542ec9e4a7824b28b0136bd5d02ba6698f59f8398cb: Status 404 returned error can't find the container with id a4a09fc53e7eef13139d8542ec9e4a7824b28b0136bd5d02ba6698f59f8398cb Oct 31 03:39:42.345237 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:42.345208 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:42.799337 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:42.799301 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-565x2" event={"ID":"2df4691e-00da-49e4-bb70-02ad5fad1625","Type":"ContainerStarted","Data":"ed99f6d7f9c9965cba8faa9b6ffd7e6e3abf7f934a34f6037c05fec82ce8394b"} Oct 31 03:39:42.799816 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:42.799368 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:39:42.800629 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:42.800604 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" event={"ID":"33c358fc-a3bb-4db1-80b3-d1f799e29621","Type":"ContainerStarted","Data":"a4a09fc53e7eef13139d8542ec9e4a7824b28b0136bd5d02ba6698f59f8398cb"} Oct 31 03:39:45.829582 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:45.829538 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" event={"ID":"33c358fc-a3bb-4db1-80b3-d1f799e29621","Type":"ContainerStarted","Data":"2ec8d80ac09b9869223283c18d704a970c1bc3d798913c226c0f8ef8b6986aaf"} Oct 31 03:39:45.935094 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:45.935059 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:45.942101 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:45.942047 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" podStartSLOduration=2.444054116 podStartE2EDuration="4.942032511s" totalImagesPullingTime="2.497978395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:45.940750839 +0000 UTC m=+682.941376067" watchObservedRunningTime="2026-10-31 03:39:45.942032511 +0000 UTC m=+682.942657717" Oct 31 03:39:48.344535 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:48.344496 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:48.357005 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:48.356979 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:48.858232 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:48.858199 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bc65b" Oct 31 03:39:49.045249 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:49.045212 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:39:51.934938 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:51.934885 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:51.947327 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:51.947300 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:52.884391 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:52.884360 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qlm9m" Oct 31 03:39:53.294420 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.294382 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh"] Oct 31 03:39:53.298231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.298212 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.347521 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.347477 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 03:39:53.365772 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.365738 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.365980 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.365787 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.365980 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.365857 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c785f917-ad62-4bcf-9b30-dcbebdbda03b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.366146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.366004 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.366146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.366103 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.366258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.366188 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms4pp\" (UniqueName: \"kubernetes.io/projected/c785f917-ad62-4bcf-9b30-dcbebdbda03b-kube-api-access-ms4pp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.440589 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.440506 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh"] Oct 31 03:39:53.467293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.467259 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.467507 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.467362 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.467507 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.467442 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4pp\" (UniqueName: \"kubernetes.io/projected/c785f917-ad62-4bcf-9b30-dcbebdbda03b-kube-api-access-ms4pp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.467650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.467505 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.467650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.467563 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.467650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.467596 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c785f917-ad62-4bcf-9b30-dcbebdbda03b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.468105 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.468078 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.468673 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.468648 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.468818 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.468783 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.470564 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.470532 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c785f917-ad62-4bcf-9b30-dcbebdbda03b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.470712 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.470688 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c785f917-ad62-4bcf-9b30-dcbebdbda03b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.526586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.526550 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4pp\" (UniqueName: \"kubernetes.io/projected/c785f917-ad62-4bcf-9b30-dcbebdbda03b-kube-api-access-ms4pp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh\" (UID: \"c785f917-ad62-4bcf-9b30-dcbebdbda03b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.611195 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.611086 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:53.810244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:53.810206 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 03:39:54.034253 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:54.034226 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh"] Oct 31 03:39:54.035787 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:39:54.035751 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc785f917_ad62_4bcf_9b30_dcbebdbda03b.slice/crio-e9e5edfe71dbe67461b71815b3a7d5c9e9613d27ad08e527597b07b228ee5876 WatchSource:0}: Error finding container e9e5edfe71dbe67461b71815b3a7d5c9e9613d27ad08e527597b07b228ee5876: Status 404 returned error can't find the container with id e9e5edfe71dbe67461b71815b3a7d5c9e9613d27ad08e527597b07b228ee5876 Oct 31 03:39:54.886871 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:54.886834 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" event={"ID":"c785f917-ad62-4bcf-9b30-dcbebdbda03b","Type":"ContainerStarted","Data":"3765bbd228a8bb942b7d0d9b58eb9fec69c0d24d6f3890ab11e98c379161356f"} Oct 31 03:39:54.887324 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:54.886901 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" event={"ID":"c785f917-ad62-4bcf-9b30-dcbebdbda03b","Type":"ContainerStarted","Data":"e9e5edfe71dbe67461b71815b3a7d5c9e9613d27ad08e527597b07b228ee5876"} Oct 31 03:39:54.998491 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:54.998378 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" podStartSLOduration=1.394978109 podStartE2EDuration="1.998362328s" totalImagesPullingTime="603.384219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:54.997252907 +0000 UTC m=+691.997878126" watchObservedRunningTime="2026-10-31 03:39:54.998362328 +0000 UTC m=+691.998987594" Oct 31 03:39:55.611844 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:55.611804 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:39:57.357396 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:39:57.357352 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:40:02.845172 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:40:02.845120 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:40:03.611593 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:03.611551 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:40:03.626530 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:03.626497 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:40:03.965328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:03.963369 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" Oct 31 03:40:17.254076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.254034 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v"] Oct 31 03:40:17.262817 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.262787 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.400736 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.400694 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.400898 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.400742 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.400898 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.400826 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec938ed-e85d-42fe-a99c-301a2b361fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.400898 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.400882 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.401012 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.400959 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.401012 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.400984 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.401012 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.401005 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.401105 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.401021 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbvb8\" (UniqueName: \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-kube-api-access-qbvb8\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.401210 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.401186 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.467280 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.467246 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v"] Oct 31 03:40:17.502081 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502043 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502087 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502115 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbvb8\" (UniqueName: \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-kube-api-access-qbvb8\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502175 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502223 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502248 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502273 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec938ed-e85d-42fe-a99c-301a2b361fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502300 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502658 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502416 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502658 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502607 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502775 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502731 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502830 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502800 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502890 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502865 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.502995 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.502975 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec938ed-e85d-42fe-a99c-301a2b361fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.504445 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.504405 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.504820 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.504799 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.560361 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.560320 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.560512 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.560453 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbvb8\" (UniqueName: \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-kube-api-access-qbvb8\") pod \"maas-default-gateway-openshift-default-7854949c9-wvs6v\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.576286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.576259 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:17.999690 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:17.999664 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v"] Oct 31 03:40:18.001043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:40:18.001016 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec938ed_e85d_42fe_a99c_301a2b361fef.slice/crio-3e5551b52169a03cdae0a1e4876da5a1ef2596cc03ebad842ddc4c73b42cc960 WatchSource:0}: Error finding container 3e5551b52169a03cdae0a1e4876da5a1ef2596cc03ebad842ddc4c73b42cc960: Status 404 returned error can't find the container with id 3e5551b52169a03cdae0a1e4876da5a1ef2596cc03ebad842ddc4c73b42cc960 Oct 31 03:40:18.003151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:18.003102 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:40:18.003207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:18.003192 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:40:18.003243 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:18.003219 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:40:18.039547 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:18.039518 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" event={"ID":"7ec938ed-e85d-42fe-a99c-301a2b361fef","Type":"ContainerStarted","Data":"3e5551b52169a03cdae0a1e4876da5a1ef2596cc03ebad842ddc4c73b42cc960"} Oct 31 03:40:19.048116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:19.048077 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" event={"ID":"7ec938ed-e85d-42fe-a99c-301a2b361fef","Type":"ContainerStarted","Data":"6e020e444d920e041f3a7d6b6f5b9762633b95c7a6058311dd47526e6355ec47"} Oct 31 03:40:19.258295 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:19.258242 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" podStartSLOduration=2.258227563 podStartE2EDuration="2.258227563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:19.211545214 +0000 UTC m=+716.212170443" watchObservedRunningTime="2026-10-31 03:40:19.258227563 +0000 UTC m=+716.258852791" Oct 31 03:40:19.576978 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:19.576935 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:19.578617 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:19.578589 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wvs6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:40:19.578761 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:19.578647 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 03:40:19.709115 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:19.709079 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:40:20.577165 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:20.577099 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wvs6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:40:20.577597 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:20.577178 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 03:40:21.576799 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:21.576746 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wvs6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:40:21.577072 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:21.576820 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 03:40:22.580146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:22.579519 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wvs6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:40:22.580146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:22.579607 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 03:40:23.577082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:23.577042 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wvs6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:40:23.577284 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:23.577150 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 03:40:24.577175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:24.577138 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wvs6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:40:24.577658 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:24.577198 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 03:40:26.578340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:26.578291 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wvs6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:26.578754 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:26.578368 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:27.619343 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:27.619302 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wvs6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:27.619946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:27.619907 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:28.496439 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:28.496400 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:28.496730 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:28.496709 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:28.497405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:28.497387 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:40:32.845915 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:40:32.845879 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:40:36.025826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:36.025785 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66fb56dd6d-wxqfb"] Oct 31 03:40:36.031216 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:36.031191 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66fb56dd6d-wxqfb" Oct 31 03:40:36.083032 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:36.082999 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66fb56dd6d-wxqfb"] Oct 31 03:40:36.134427 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:36.134387 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e9f99a4-7d1c-4783-9c5f-d565dd5f0931-tls-cert\") pod \"authorino-66fb56dd6d-wxqfb\" (UID: \"2e9f99a4-7d1c-4783-9c5f-d565dd5f0931\") " pod="kuadrant-system/authorino-66fb56dd6d-wxqfb" Oct 31 03:40:36.134613 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:36.134456 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7g4x\" (UniqueName: \"kubernetes.io/projected/2e9f99a4-7d1c-4783-9c5f-d565dd5f0931-kube-api-access-h7g4x\") pod \"authorino-66fb56dd6d-wxqfb\" (UID: \"2e9f99a4-7d1c-4783-9c5f-d565dd5f0931\") " pod="kuadrant-system/authorino-66fb56dd6d-wxqfb" Oct 31 03:40:36.235999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:36.235955 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7g4x\" (UniqueName: \"kubernetes.io/projected/2e9f99a4-7d1c-4783-9c5f-d565dd5f0931-kube-api-access-h7g4x\") pod \"authorino-66fb56dd6d-wxqfb\" (UID: \"2e9f99a4-7d1c-4783-9c5f-d565dd5f0931\") " pod="kuadrant-system/authorino-66fb56dd6d-wxqfb" Oct 31 03:40:36.236226 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:36.236046 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e9f99a4-7d1c-4783-9c5f-d565dd5f0931-tls-cert\") pod \"authorino-66fb56dd6d-wxqfb\" (UID: \"2e9f99a4-7d1c-4783-9c5f-d565dd5f0931\") " pod="kuadrant-system/authorino-66fb56dd6d-wxqfb" Oct 31 03:40:36.238540 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:36.238515 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e9f99a4-7d1c-4783-9c5f-d565dd5f0931-tls-cert\") pod \"authorino-66fb56dd6d-wxqfb\" (UID: \"2e9f99a4-7d1c-4783-9c5f-d565dd5f0931\") " pod="kuadrant-system/authorino-66fb56dd6d-wxqfb" Oct 31 03:40:36.293976 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:36.293911 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7g4x\" (UniqueName: \"kubernetes.io/projected/2e9f99a4-7d1c-4783-9c5f-d565dd5f0931-kube-api-access-h7g4x\") pod \"authorino-66fb56dd6d-wxqfb\" (UID: \"2e9f99a4-7d1c-4783-9c5f-d565dd5f0931\") " pod="kuadrant-system/authorino-66fb56dd6d-wxqfb" Oct 31 03:40:36.343577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:36.343526 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66fb56dd6d-wxqfb" Oct 31 03:40:36.710094 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:36.710070 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66fb56dd6d-wxqfb"] Oct 31 03:40:36.711887 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:40:36.711862 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e9f99a4_7d1c_4783_9c5f_d565dd5f0931.slice/crio-9f987faa563052751410007305fd93a179b84539cef2016331bd5ed2f99c97de WatchSource:0}: Error finding container 9f987faa563052751410007305fd93a179b84539cef2016331bd5ed2f99c97de: Status 404 returned error can't find the container with id 9f987faa563052751410007305fd93a179b84539cef2016331bd5ed2f99c97de Oct 31 03:40:37.172752 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:37.172713 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66fb56dd6d-wxqfb" event={"ID":"2e9f99a4-7d1c-4783-9c5f-d565dd5f0931","Type":"ContainerStarted","Data":"9f987faa563052751410007305fd93a179b84539cef2016331bd5ed2f99c97de"} Oct 31 03:40:39.185483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:39.185441 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66fb56dd6d-wxqfb" event={"ID":"2e9f99a4-7d1c-4783-9c5f-d565dd5f0931","Type":"ContainerStarted","Data":"26259056c84de6609c4b09989fea7425307d1d12d9d128554caac4907f9f299d"} Oct 31 03:40:39.295328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:39.295262 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-66fb56dd6d-wxqfb" podStartSLOduration=2.320033091 podStartE2EDuration="4.295242362s" totalImagesPullingTime="1.975209271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:39.29326963 +0000 UTC m=+736.293894859" watchObservedRunningTime="2026-10-31 03:40:39.295242362 +0000 UTC m=+736.295867591" Oct 31 03:40:39.323999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:39.323965 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78db76d6d7-vt4tz"] Oct 31 03:40:39.324209 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:39.324186 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-78db76d6d7-vt4tz" podUID="883eeb42-2686-4880-87a0-650aa653e124" containerName="authorino" containerID="cri-o://2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52" gracePeriod=30 Oct 31 03:40:39.590376 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:39.590348 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78db76d6d7-vt4tz" Oct 31 03:40:39.664201 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:39.664166 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883eeb42-2686-4880-87a0-650aa653e124-kube-api-access-nklh4\" (UniqueName: \"kubernetes.io/projected/883eeb42-2686-4880-87a0-650aa653e124-kube-api-access-nklh4\") pod \"883eeb42-2686-4880-87a0-650aa653e124\" (UID: \"883eeb42-2686-4880-87a0-650aa653e124\") " Oct 31 03:40:39.664405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:39.664271 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/883eeb42-2686-4880-87a0-650aa653e124-tls-cert\" (UniqueName: \"kubernetes.io/secret/883eeb42-2686-4880-87a0-650aa653e124-tls-cert\") pod \"883eeb42-2686-4880-87a0-650aa653e124\" (UID: \"883eeb42-2686-4880-87a0-650aa653e124\") " Oct 31 03:40:39.666354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:39.666320 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883eeb42-2686-4880-87a0-650aa653e124-kube-api-access-nklh4" pod "883eeb42-2686-4880-87a0-650aa653e124" (UID: "883eeb42-2686-4880-87a0-650aa653e124"). InnerVolumeSpecName "kube-api-access-nklh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:39.674833 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:39.674791 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883eeb42-2686-4880-87a0-650aa653e124-tls-cert" pod "883eeb42-2686-4880-87a0-650aa653e124" (UID: "883eeb42-2686-4880-87a0-650aa653e124"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:39.765527 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:39.765487 4426 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/883eeb42-2686-4880-87a0-650aa653e124-tls-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:39.765527 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:39.765518 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nklh4\" (UniqueName: \"kubernetes.io/projected/883eeb42-2686-4880-87a0-650aa653e124-kube-api-access-nklh4\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:40.192776 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:40.192734 4426 generic.go:356] "Generic (PLEG): container finished" podID="883eeb42-2686-4880-87a0-650aa653e124" containerID="2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52" exitCode=0 Oct 31 03:40:40.192776 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:40.192781 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78db76d6d7-vt4tz" Oct 31 03:40:40.193353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:40.192818 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78db76d6d7-vt4tz" event={"ID":"883eeb42-2686-4880-87a0-650aa653e124","Type":"ContainerDied","Data":"2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52"} Oct 31 03:40:40.193353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:40.192877 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78db76d6d7-vt4tz" event={"ID":"883eeb42-2686-4880-87a0-650aa653e124","Type":"ContainerDied","Data":"e99db5d71351f49d48655048d3ae40e98d2f60de7155e80b0c40ba8c5ec0decd"} Oct 31 03:40:40.193353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:40.192907 4426 scope.go:122] "RemoveContainer" containerID="2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52" Oct 31 03:40:40.202697 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:40.202676 4426 scope.go:122] "RemoveContainer" containerID="2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52" Oct 31 03:40:40.203029 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:40:40.203006 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52\": container with ID starting with 2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52 not found: ID does not exist" containerID="2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52" Oct 31 03:40:40.203077 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:40.203039 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52"} err="failed to get container status \"2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52\": rpc error: code = NotFound desc = could not find container \"2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52\": container with ID starting with 2a5ed9f5d3e4d03522344f3d54b938524c4436b5ee797e6054b69da4e29b2c52 not found: ID does not exist" Oct 31 03:40:40.347757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:40.347718 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78db76d6d7-vt4tz"] Oct 31 03:40:40.351640 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:40.351609 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78db76d6d7-vt4tz"] Oct 31 03:40:41.129648 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:41.129589 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="883eeb42-2686-4880-87a0-650aa653e124" path="/var/lib/kubelet/pods/883eeb42-2686-4880-87a0-650aa653e124/volumes" Oct 31 03:40:46.315724 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.315689 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g"] Oct 31 03:40:46.316174 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.316157 4426 cpu_manager.go:414] "containerMap: removing container" podUID="883eeb42-2686-4880-87a0-650aa653e124" containerName="authorino" Oct 31 03:40:46.316250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.316175 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883eeb42-2686-4880-87a0-650aa653e124" containerName="authorino" Oct 31 03:40:46.316293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.316260 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="883eeb42-2686-4880-87a0-650aa653e124" containerName="authorino" Oct 31 03:40:46.320870 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.320839 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:40:46.370148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.370091 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 03:40:46.416431 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.416395 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 03:40:46.416630 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.416455 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-9lwx4\"" Oct 31 03:40:46.420868 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.420835 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g"] Oct 31 03:40:46.437779 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.437754 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-hdl8g\" (UID: \"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:40:46.437925 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.437811 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw995\" (UniqueName: \"kubernetes.io/projected/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-kube-api-access-qw995\") pod \"ai-gateway-controller-5d9cd99f45-hdl8g\" (UID: \"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:40:46.538487 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.538434 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-hdl8g\" (UID: \"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:40:46.538678 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.538516 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw995\" (UniqueName: \"kubernetes.io/projected/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-kube-api-access-qw995\") pod \"ai-gateway-controller-5d9cd99f45-hdl8g\" (UID: \"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:40:46.540978 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.540946 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-hdl8g\" (UID: \"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:40:46.595928 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.595852 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw995\" (UniqueName: \"kubernetes.io/projected/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-kube-api-access-qw995\") pod \"ai-gateway-controller-5d9cd99f45-hdl8g\" (UID: \"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:40:46.634744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:46.634712 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:40:47.000759 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:47.000731 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g"] Oct 31 03:40:47.002031 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:40:47.002004 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c66ffa_d6b7_4c8e_be7f_7749896fdfd4.slice/crio-90d1fa308495723a0a24212b483c71674651460336bf3bdd28e1e1b4385ec563 WatchSource:0}: Error finding container 90d1fa308495723a0a24212b483c71674651460336bf3bdd28e1e1b4385ec563: Status 404 returned error can't find the container with id 90d1fa308495723a0a24212b483c71674651460336bf3bdd28e1e1b4385ec563 Oct 31 03:40:47.246761 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:47.246722 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" event={"ID":"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4","Type":"ContainerStarted","Data":"90d1fa308495723a0a24212b483c71674651460336bf3bdd28e1e1b4385ec563"} Oct 31 03:40:49.260679 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:49.260639 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" event={"ID":"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4","Type":"ContainerStarted","Data":"2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170"} Oct 31 03:40:49.261178 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:49.260844 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:40:49.370071 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:49.370009 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" podStartSLOduration=1.230956935 podStartE2EDuration="3.369993908s" totalImagesPullingTime="2.139036973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:49.369524467 +0000 UTC m=+746.370149716" watchObservedRunningTime="2026-10-31 03:40:49.369993908 +0000 UTC m=+746.370619137" Oct 31 03:40:53.341869 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.341828 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv"] Oct 31 03:40:53.345933 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.345908 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" Oct 31 03:40:53.401939 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.401907 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv"] Oct 31 03:40:53.444024 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.443993 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bpzzr\"" Oct 31 03:40:53.444220 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.444064 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 03:40:53.506742 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.506704 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9wmqv\" (UID: \"a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" Oct 31 03:40:53.506909 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.506821 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8247\" (UniqueName: \"kubernetes.io/projected/a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f-kube-api-access-z8247\") pod \"payload-pre-processing-7b98bb678c-9wmqv\" (UID: \"a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" Oct 31 03:40:53.507198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.507174 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-97mff"] Oct 31 03:40:53.510855 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.510839 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" Oct 31 03:40:53.567629 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.567594 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-97mff"] Oct 31 03:40:53.607499 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.607406 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9wmqv\" (UID: \"a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" Oct 31 03:40:53.607499 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.607455 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8247\" (UniqueName: \"kubernetes.io/projected/a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f-kube-api-access-z8247\") pod \"payload-pre-processing-7b98bb678c-9wmqv\" (UID: \"a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" Oct 31 03:40:53.607499 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.607480 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvg9\" (UniqueName: \"kubernetes.io/projected/e44d8d56-e743-4335-a287-2330fdeeb3a1-kube-api-access-zpvg9\") pod \"payload-processing-55fbbbf9dc-97mff\" (UID: \"e44d8d56-e743-4335-a287-2330fdeeb3a1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" Oct 31 03:40:53.607499 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.607499 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44d8d56-e743-4335-a287-2330fdeeb3a1-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-97mff\" (UID: \"e44d8d56-e743-4335-a287-2330fdeeb3a1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" Oct 31 03:40:53.608212 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.608190 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9wmqv\" (UID: \"a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" Oct 31 03:40:53.664946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.664907 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8247\" (UniqueName: \"kubernetes.io/projected/a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f-kube-api-access-z8247\") pod \"payload-pre-processing-7b98bb678c-9wmqv\" (UID: \"a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" Oct 31 03:40:53.708867 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.708836 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvg9\" (UniqueName: \"kubernetes.io/projected/e44d8d56-e743-4335-a287-2330fdeeb3a1-kube-api-access-zpvg9\") pod \"payload-processing-55fbbbf9dc-97mff\" (UID: \"e44d8d56-e743-4335-a287-2330fdeeb3a1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" Oct 31 03:40:53.709040 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.708871 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44d8d56-e743-4335-a287-2330fdeeb3a1-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-97mff\" (UID: \"e44d8d56-e743-4335-a287-2330fdeeb3a1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" Oct 31 03:40:53.709612 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.709581 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44d8d56-e743-4335-a287-2330fdeeb3a1-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-97mff\" (UID: \"e44d8d56-e743-4335-a287-2330fdeeb3a1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" Oct 31 03:40:53.766455 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.766420 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvg9\" (UniqueName: \"kubernetes.io/projected/e44d8d56-e743-4335-a287-2330fdeeb3a1-kube-api-access-zpvg9\") pod \"payload-processing-55fbbbf9dc-97mff\" (UID: \"e44d8d56-e743-4335-a287-2330fdeeb3a1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" Oct 31 03:40:53.825991 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.825960 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" Oct 31 03:40:53.960343 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:53.960313 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" Oct 31 03:40:54.194349 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:54.194319 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-97mff"] Oct 31 03:40:54.195843 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:40:54.195811 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44d8d56_e743_4335_a287_2330fdeeb3a1.slice/crio-1828e926d8b7d70e3ca95f8e77117054bb03dc682cda61ce83f9ca32a30ea55c WatchSource:0}: Error finding container 1828e926d8b7d70e3ca95f8e77117054bb03dc682cda61ce83f9ca32a30ea55c: Status 404 returned error can't find the container with id 1828e926d8b7d70e3ca95f8e77117054bb03dc682cda61ce83f9ca32a30ea55c Oct 31 03:40:54.291520 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:54.291483 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" event={"ID":"e44d8d56-e743-4335-a287-2330fdeeb3a1","Type":"ContainerStarted","Data":"1828e926d8b7d70e3ca95f8e77117054bb03dc682cda61ce83f9ca32a30ea55c"} Oct 31 03:40:54.326068 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:54.326039 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv"] Oct 31 03:40:54.327538 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:40:54.327512 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dcf8e8_043d_48d9_9c2e_cdb93c878b5f.slice/crio-59c6fb67ffb233a9a16da2861e76cb60c74b2cac07eb8c9d3c78120a2b65351c WatchSource:0}: Error finding container 59c6fb67ffb233a9a16da2861e76cb60c74b2cac07eb8c9d3c78120a2b65351c: Status 404 returned error can't find the container with id 59c6fb67ffb233a9a16da2861e76cb60c74b2cac07eb8c9d3c78120a2b65351c Oct 31 03:40:55.298192 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:55.298157 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" event={"ID":"a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f","Type":"ContainerStarted","Data":"59c6fb67ffb233a9a16da2861e76cb60c74b2cac07eb8c9d3c78120a2b65351c"} Oct 31 03:40:58.320639 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:58.320588 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" event={"ID":"e44d8d56-e743-4335-a287-2330fdeeb3a1","Type":"ContainerStarted","Data":"afd9585020bd7bb460e65807f10bc06bf5403374837e223e1d24bfca287e5b0b"} Oct 31 03:40:58.321231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:58.320689 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" Oct 31 03:40:58.322299 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:58.322272 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" event={"ID":"a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f","Type":"ContainerStarted","Data":"650bb9879a92941f615934ca5902df48ab51d59b801b9c1ad64de414324f5c5d"} Oct 31 03:40:58.322402 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:58.322331 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" Oct 31 03:40:58.435271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:58.435216 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" podStartSLOduration=2.226408172 podStartE2EDuration="5.435200809s" totalImagesPullingTime="3.208792637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:58.432893098 +0000 UTC m=+755.433518326" watchObservedRunningTime="2026-10-31 03:40:58.435200809 +0000 UTC m=+755.435826037" Oct 31 03:40:58.543281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:40:58.543228 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" podStartSLOduration=2.476591762 podStartE2EDuration="5.543203856s" totalImagesPullingTime="3.066612094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:58.541674743 +0000 UTC m=+755.542299970" watchObservedRunningTime="2026-10-31 03:40:58.543203856 +0000 UTC m=+755.543829083" Oct 31 03:41:00.269212 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:00.269183 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:41:00.623833 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:00.623753 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-57d9f499dc-7th7b"] Oct 31 03:41:00.627715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:00.627696 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" Oct 31 03:41:00.682901 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:00.682872 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-57d9f499dc-7th7b"] Oct 31 03:41:00.773229 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:00.773192 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkkzs\" (UniqueName: \"kubernetes.io/projected/41eb1d0e-8fed-41d8-99a1-91c359891566-kube-api-access-bkkzs\") pod \"ai-gateway-controller-57d9f499dc-7th7b\" (UID: \"41eb1d0e-8fed-41d8-99a1-91c359891566\") " pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" Oct 31 03:41:00.773410 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:00.773241 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41eb1d0e-8fed-41d8-99a1-91c359891566-metrics-certs\") pod \"ai-gateway-controller-57d9f499dc-7th7b\" (UID: \"41eb1d0e-8fed-41d8-99a1-91c359891566\") " pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" Oct 31 03:41:00.874325 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:00.874227 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkzs\" (UniqueName: \"kubernetes.io/projected/41eb1d0e-8fed-41d8-99a1-91c359891566-kube-api-access-bkkzs\") pod \"ai-gateway-controller-57d9f499dc-7th7b\" (UID: \"41eb1d0e-8fed-41d8-99a1-91c359891566\") " pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" Oct 31 03:41:00.874325 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:00.874275 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41eb1d0e-8fed-41d8-99a1-91c359891566-metrics-certs\") pod \"ai-gateway-controller-57d9f499dc-7th7b\" (UID: \"41eb1d0e-8fed-41d8-99a1-91c359891566\") " pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" Oct 31 03:41:00.876736 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:00.876716 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41eb1d0e-8fed-41d8-99a1-91c359891566-metrics-certs\") pod \"ai-gateway-controller-57d9f499dc-7th7b\" (UID: \"41eb1d0e-8fed-41d8-99a1-91c359891566\") " pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" Oct 31 03:41:00.931641 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:00.931608 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkkzs\" (UniqueName: \"kubernetes.io/projected/41eb1d0e-8fed-41d8-99a1-91c359891566-kube-api-access-bkkzs\") pod \"ai-gateway-controller-57d9f499dc-7th7b\" (UID: \"41eb1d0e-8fed-41d8-99a1-91c359891566\") " pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" Oct 31 03:41:00.940489 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:00.940460 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" Oct 31 03:41:01.310975 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:01.310932 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-57d9f499dc-7th7b"] Oct 31 03:41:01.316324 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:41:01.316289 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41eb1d0e_8fed_41d8_99a1_91c359891566.slice/crio-33b014f8f593ce183927ac6401e4a51d528f30781ed67328d78f185ac45cd0bf WatchSource:0}: Error finding container 33b014f8f593ce183927ac6401e4a51d528f30781ed67328d78f185ac45cd0bf: Status 404 returned error can't find the container with id 33b014f8f593ce183927ac6401e4a51d528f30781ed67328d78f185ac45cd0bf Oct 31 03:41:01.341196 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:01.341159 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" event={"ID":"41eb1d0e-8fed-41d8-99a1-91c359891566","Type":"ContainerStarted","Data":"33b014f8f593ce183927ac6401e4a51d528f30781ed67328d78f185ac45cd0bf"} Oct 31 03:41:02.349609 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:02.349553 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" event={"ID":"41eb1d0e-8fed-41d8-99a1-91c359891566","Type":"ContainerStarted","Data":"e393a3d5b9b0a1c0013bca77407e9a0edf81be1950ade6da094eb1ac03f4ee4b"} Oct 31 03:41:02.350018 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:02.349730 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" Oct 31 03:41:02.460336 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:02.460224 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" podStartSLOduration=1.570797426 podStartE2EDuration="2.460204789s" totalImagesPullingTime="889.407363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:02.458367222 +0000 UTC m=+759.458992449" watchObservedRunningTime="2026-10-31 03:41:02.460204789 +0000 UTC m=+759.460830037" Oct 31 03:41:02.845182 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:41:02.845079 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:41:09.332098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:09.332022 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-97mff" Oct 31 03:41:09.332098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:09.332091 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9wmqv" Oct 31 03:41:13.358231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:13.358200 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-57d9f499dc-7th7b" Oct 31 03:41:13.505145 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:13.505098 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g"] Oct 31 03:41:13.505398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:13.505375 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" podUID="04c66ffa-d6b7-4c8e-be7f-7749896fdfd4" containerName="manager" containerID="cri-o://2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170" gracePeriod=30 Oct 31 03:41:13.763459 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:13.763430 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:41:13.784681 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:13.784654 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-kube-api-access-qw995\" (UniqueName: \"kubernetes.io/projected/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-kube-api-access-qw995\") pod \"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4\" (UID: \"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4\") " Oct 31 03:41:13.784820 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:13.784717 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-metrics-certs\") pod \"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4\" (UID: \"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4\") " Oct 31 03:41:13.786822 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:13.786795 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-kube-api-access-qw995" pod "04c66ffa-d6b7-4c8e-be7f-7749896fdfd4" (UID: "04c66ffa-d6b7-4c8e-be7f-7749896fdfd4"). InnerVolumeSpecName "kube-api-access-qw995". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:13.786822 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:13.786810 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-metrics-certs" pod "04c66ffa-d6b7-4c8e-be7f-7749896fdfd4" (UID: "04c66ffa-d6b7-4c8e-be7f-7749896fdfd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:13.886653 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:13.886562 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw995\" (UniqueName: \"kubernetes.io/projected/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-kube-api-access-qw995\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:13.886653 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:13.886615 4426 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4-metrics-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:14.419454 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:14.419421 4426 generic.go:356] "Generic (PLEG): container finished" podID="04c66ffa-d6b7-4c8e-be7f-7749896fdfd4" containerID="2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170" exitCode=0 Oct 31 03:41:14.419858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:14.419477 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" Oct 31 03:41:14.419858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:14.419506 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" event={"ID":"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4","Type":"ContainerDied","Data":"2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170"} Oct 31 03:41:14.419858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:14.419568 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g" event={"ID":"04c66ffa-d6b7-4c8e-be7f-7749896fdfd4","Type":"ContainerDied","Data":"90d1fa308495723a0a24212b483c71674651460336bf3bdd28e1e1b4385ec563"} Oct 31 03:41:14.419858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:14.419599 4426 scope.go:122] "RemoveContainer" containerID="2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170" Oct 31 03:41:14.429418 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:14.429397 4426 scope.go:122] "RemoveContainer" containerID="2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170" Oct 31 03:41:14.429681 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:41:14.429663 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170\": container with ID starting with 2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170 not found: ID does not exist" containerID="2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170" Oct 31 03:41:14.429726 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:14.429691 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170"} err="failed to get container status \"2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170\": rpc error: code = NotFound desc = could not find container \"2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170\": container with ID starting with 2d208bdcf6712a7ed6c5521e38bd1266b1cd7bb56ea7e06d6d6d35c3c5d8b170 not found: ID does not exist" Oct 31 03:41:14.574426 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:14.574385 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g"] Oct 31 03:41:14.620590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:14.620558 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hdl8g"] Oct 31 03:41:15.128540 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:41:15.128504 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c66ffa-d6b7-4c8e-be7f-7749896fdfd4" path="/var/lib/kubelet/pods/04c66ffa-d6b7-4c8e-be7f-7749896fdfd4/volumes" Oct 31 03:41:32.845195 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:41:32.845159 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:42:02.844911 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:42:02.844869 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:42:32.845142 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:42:32.845100 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:43:02.845113 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:43:02.845073 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:43:03.477974 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.477935 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x"] Oct 31 03:43:03.478388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.478375 4426 cpu_manager.go:414] "containerMap: removing container" podUID="04c66ffa-d6b7-4c8e-be7f-7749896fdfd4" containerName="manager" Oct 31 03:43:03.478440 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.478389 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c66ffa-d6b7-4c8e-be7f-7749896fdfd4" containerName="manager" Oct 31 03:43:03.478475 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.478466 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c66ffa-d6b7-4c8e-be7f-7749896fdfd4" containerName="manager" Oct 31 03:43:03.481595 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.481574 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.555088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.555052 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x"] Oct 31 03:43:03.576864 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.576822 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-537e34-openshift-default-dockercfg-5x4wl\"" Oct 31 03:43:03.614682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.614626 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-envoy\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.614682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.614681 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnkb\" (UniqueName: \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-kube-api-access-clnkb\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.614968 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.614835 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-certs\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.614968 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.614926 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-token\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.614968 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.614962 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-data\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.615194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.615047 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-credential-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.615194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.615065 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istiod-ca-cert\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.615194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.615140 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.615345 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.615218 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-podinfo\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716075 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716031 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716075 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716081 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-podinfo\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716383 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716163 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-envoy\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716383 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716201 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnkb\" (UniqueName: \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-kube-api-access-clnkb\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716383 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716266 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-certs\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716383 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716303 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-token\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716383 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716331 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-data\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716383 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716368 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-credential-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716394 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istiod-ca-cert\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716511 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716803 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716725 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-certs\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716860 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716796 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-data\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.716860 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.716820 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-credential-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.717095 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.717072 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istiod-ca-cert\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.718592 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.718559 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-envoy\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.718738 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.718697 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-podinfo\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.779395 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.779307 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-token\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.779395 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.779310 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnkb\" (UniqueName: \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-kube-api-access-clnkb\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:03.797225 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:03.797187 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:04.136608 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.136509 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm"] Oct 31 03:43:04.140705 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.140680 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.189778 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.189745 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-412ba5-openshift-default-dockercfg-dglp9\"" Oct 31 03:43:04.207504 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.207472 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm"] Oct 31 03:43:04.221573 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.221537 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-certs\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.221573 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.221581 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-credential-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.221807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.221605 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-token\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.221851 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.221825 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-envoy\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.221903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.221879 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.221995 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.221967 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-podinfo\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.222071 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.222049 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85150ab8-f8d1-40d2-b540-0f65db33199b-istiod-ca-cert\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.222143 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.222104 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4wpf\" (UniqueName: \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-kube-api-access-k4wpf\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.222255 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.222230 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-data\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.244272 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.244233 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x"] Oct 31 03:43:04.256613 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:43:04.256580 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f0fec9_03b7_4707_8d26_f40f5d1d312c.slice/crio-4afcd549d63d3bcfab98417d4f79535a3530e702ba778cbdca64eedfcd3a14bf WatchSource:0}: Error finding container 4afcd549d63d3bcfab98417d4f79535a3530e702ba778cbdca64eedfcd3a14bf: Status 404 returned error can't find the container with id 4afcd549d63d3bcfab98417d4f79535a3530e702ba778cbdca64eedfcd3a14bf Oct 31 03:43:04.258936 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.258905 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:04.259021 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.258971 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:04.259021 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.258997 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:04.322757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.322716 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-data\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.322757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.322759 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-certs\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323000 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.322784 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-credential-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323000 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.322807 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-token\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323000 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.322950 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-envoy\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323000 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.322998 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323254 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.323057 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-podinfo\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323254 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.323097 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85150ab8-f8d1-40d2-b540-0f65db33199b-istiod-ca-cert\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323254 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.323163 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-data\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323254 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.323172 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4wpf\" (UniqueName: \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-kube-api-access-k4wpf\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323254 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.323201 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-credential-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323509 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.323421 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-certs\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323509 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.323484 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.323982 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.323960 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85150ab8-f8d1-40d2-b540-0f65db33199b-istiod-ca-cert\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.325709 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.325625 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-podinfo\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.325817 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.325795 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-envoy\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.384196 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.384097 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-token\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.393413 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.393373 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4wpf\" (UniqueName: \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-kube-api-access-k4wpf\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.454609 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.454565 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:04.796437 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.796354 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8"] Oct 31 03:43:04.801414 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.801389 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:04.851285 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.851243 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a64fd2-openshift-default-dockercfg-2llsl\"" Oct 31 03:43:04.860777 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.860741 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8"] Oct 31 03:43:04.904311 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.904277 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm"] Oct 31 03:43:04.906534 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:43:04.906501 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85150ab8_f8d1_40d2_b540_0f65db33199b.slice/crio-34e05b09397dcb03b1e9863c7f9cd426b230c3cd915021c081f15f27f9c6b3d0 WatchSource:0}: Error finding container 34e05b09397dcb03b1e9863c7f9cd426b230c3cd915021c081f15f27f9c6b3d0: Status 404 returned error can't find the container with id 34e05b09397dcb03b1e9863c7f9cd426b230c3cd915021c081f15f27f9c6b3d0 Oct 31 03:43:04.909238 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.909198 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:04.909348 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.909289 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:04.909348 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.909327 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:04.934836 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.934795 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-certs\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:04.934979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.934850 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:04.934979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.934948 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-token\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:04.935055 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.934983 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8f0271-7150-400d-bd05-da5fa011a1cf-istiod-ca-cert\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:04.935087 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.935067 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-podinfo\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:04.935154 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.935115 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-data\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:04.935217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.935202 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-credential-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:04.935258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.935225 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-envoy\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:04.935293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:04.935269 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmxh\" (UniqueName: \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-kube-api-access-rkmxh\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.036516 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.036470 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-certs\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.036714 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.036526 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.037060 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.036678 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-token\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.037164 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.037116 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8f0271-7150-400d-bd05-da5fa011a1cf-istiod-ca-cert\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.037254 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.037225 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-podinfo\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.037308 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.037284 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-data\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.037362 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.037349 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-credential-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.037424 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.037377 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-envoy\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.037483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.037421 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmxh\" (UniqueName: \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-kube-api-access-rkmxh\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.037796 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.037760 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8f0271-7150-400d-bd05-da5fa011a1cf-istiod-ca-cert\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.037796 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.037019 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-certs\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.038181 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.036959 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.038181 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.038078 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-data\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.038636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.038613 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-credential-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.040648 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.040599 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-podinfo\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.042356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.042334 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-envoy\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.054180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.054002 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" event={"ID":"f8f0fec9-03b7-4707-8d26-f40f5d1d312c","Type":"ContainerStarted","Data":"a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760"} Oct 31 03:43:05.054430 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.054354 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" event={"ID":"f8f0fec9-03b7-4707-8d26-f40f5d1d312c","Type":"ContainerStarted","Data":"4afcd549d63d3bcfab98417d4f79535a3530e702ba778cbdca64eedfcd3a14bf"} Oct 31 03:43:05.056392 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.056363 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" event={"ID":"85150ab8-f8d1-40d2-b540-0f65db33199b","Type":"ContainerStarted","Data":"ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675"} Oct 31 03:43:05.056620 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.056593 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" event={"ID":"85150ab8-f8d1-40d2-b540-0f65db33199b","Type":"ContainerStarted","Data":"34e05b09397dcb03b1e9863c7f9cd426b230c3cd915021c081f15f27f9c6b3d0"} Oct 31 03:43:05.094396 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.094364 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-token\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.094717 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.094695 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmxh\" (UniqueName: \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-kube-api-access-rkmxh\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.116578 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.116534 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:05.206555 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.206479 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" podStartSLOduration=2.206461827 podStartE2EDuration="2.206461827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:05.203916707 +0000 UTC m=+882.204541936" watchObservedRunningTime="2026-10-31 03:43:05.206461827 +0000 UTC m=+882.207087056" Oct 31 03:43:05.325415 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.325290 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" podStartSLOduration=1.325268767 podStartE2EDuration="1.325268767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:05.32128019 +0000 UTC m=+882.321905420" watchObservedRunningTime="2026-10-31 03:43:05.325268767 +0000 UTC m=+882.325893995" Oct 31 03:43:05.455380 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.455345 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:05.460283 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.460257 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:05.491236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.491208 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8"] Oct 31 03:43:05.492757 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:43:05.492730 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8f0271_7150_400d_bd05_da5fa011a1cf.slice/crio-320f57cc75ec4ffe5e2a50cd19ff6cbf9c222d55c22aa35652c01891865668d7 WatchSource:0}: Error finding container 320f57cc75ec4ffe5e2a50cd19ff6cbf9c222d55c22aa35652c01891865668d7: Status 404 returned error can't find the container with id 320f57cc75ec4ffe5e2a50cd19ff6cbf9c222d55c22aa35652c01891865668d7 Oct 31 03:43:05.495182 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.495147 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:05.495283 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.495216 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:05.495283 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.495246 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:05.797927 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.797899 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:05.803398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:05.803372 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:06.063952 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:06.063853 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" event={"ID":"5a8f0271-7150-400d-bd05-da5fa011a1cf","Type":"ContainerStarted","Data":"991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889"} Oct 31 03:43:06.063952 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:06.063917 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" event={"ID":"5a8f0271-7150-400d-bd05-da5fa011a1cf","Type":"ContainerStarted","Data":"320f57cc75ec4ffe5e2a50cd19ff6cbf9c222d55c22aa35652c01891865668d7"} Oct 31 03:43:06.064431 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:06.064401 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:06.064545 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:06.064458 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:06.065201 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:06.065180 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:06.065304 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:06.065229 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:06.117387 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:06.117351 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:06.122765 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:06.122734 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:06.178197 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:06.178108 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" podStartSLOduration=2.178091331 podStartE2EDuration="2.178091331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:06.176874532 +0000 UTC m=+883.177499763" watchObservedRunningTime="2026-10-31 03:43:06.178091331 +0000 UTC m=+883.178716558" Oct 31 03:43:07.070495 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:07.070451 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:07.071281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:07.071258 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:43:11.950174 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:11.950141 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws"] Oct 31 03:43:11.954024 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:11.954002 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 03:43:12.051776 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:12.051733 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-537e34\"" Oct 31 03:43:12.051965 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:12.051842 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-537e34-dockercfg-hsttt\"" Oct 31 03:43:12.054682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:12.054654 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws"] Oct 31 03:43:12.117161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:12.117101 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjnmn\" (UniqueName: \"kubernetes.io/projected/91986eb8-661e-4eb2-8c70-077aa1ed61b8-kube-api-access-jjnmn\") pod \"payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws\" (UID: \"91986eb8-661e-4eb2-8c70-077aa1ed61b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 03:43:12.117330 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:12.117275 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91986eb8-661e-4eb2-8c70-077aa1ed61b8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws\" (UID: \"91986eb8-661e-4eb2-8c70-077aa1ed61b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 03:43:12.218915 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:12.218806 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjnmn\" (UniqueName: \"kubernetes.io/projected/91986eb8-661e-4eb2-8c70-077aa1ed61b8-kube-api-access-jjnmn\") pod \"payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws\" (UID: \"91986eb8-661e-4eb2-8c70-077aa1ed61b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 03:43:12.219095 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:12.218938 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91986eb8-661e-4eb2-8c70-077aa1ed61b8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws\" (UID: \"91986eb8-661e-4eb2-8c70-077aa1ed61b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 03:43:12.219724 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:12.219703 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91986eb8-661e-4eb2-8c70-077aa1ed61b8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws\" (UID: \"91986eb8-661e-4eb2-8c70-077aa1ed61b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 03:43:12.278147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:12.278104 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjnmn\" (UniqueName: \"kubernetes.io/projected/91986eb8-661e-4eb2-8c70-077aa1ed61b8-kube-api-access-jjnmn\") pod \"payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws\" (UID: \"91986eb8-661e-4eb2-8c70-077aa1ed61b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 03:43:12.568714 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:12.568627 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 03:43:12.942991 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:12.942957 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws"] Oct 31 03:43:12.943792 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:43:12.943747 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91986eb8_661e_4eb2_8c70_077aa1ed61b8.slice/crio-d64c68df14b7b44d7ba03e88053b90e054a707a1298ac1196130961cded272e9 WatchSource:0}: Error finding container d64c68df14b7b44d7ba03e88053b90e054a707a1298ac1196130961cded272e9: Status 404 returned error can't find the container with id d64c68df14b7b44d7ba03e88053b90e054a707a1298ac1196130961cded272e9 Oct 31 03:43:13.106361 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:13.106322 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" event={"ID":"91986eb8-661e-4eb2-8c70-077aa1ed61b8","Type":"ContainerStarted","Data":"fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12"} Oct 31 03:43:13.106766 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:13.106390 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" event={"ID":"91986eb8-661e-4eb2-8c70-077aa1ed61b8","Type":"ContainerStarted","Data":"d64c68df14b7b44d7ba03e88053b90e054a707a1298ac1196130961cded272e9"} Oct 31 03:43:13.106766 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:13.106413 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 03:43:13.227007 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:13.226938 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" podStartSLOduration=2.226917205 podStartE2EDuration="2.226917205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:13.218618521 +0000 UTC m=+890.219243750" watchObservedRunningTime="2026-10-31 03:43:13.226917205 +0000 UTC m=+890.227542433" Oct 31 03:43:16.376786 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.376750 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl"] Oct 31 03:43:16.381018 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.380992 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 03:43:16.459100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.459060 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259d0ab8-20b4-4c18-9876-962926e794b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl\" (UID: \"259d0ab8-20b4-4c18-9876-962926e794b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 03:43:16.459320 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.459229 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qjfh\" (UniqueName: \"kubernetes.io/projected/259d0ab8-20b4-4c18-9876-962926e794b9-kube-api-access-8qjfh\") pod \"payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl\" (UID: \"259d0ab8-20b4-4c18-9876-962926e794b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 03:43:16.479459 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.479416 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-412ba5\"" Oct 31 03:43:16.479638 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.479497 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-412ba5-dockercfg-qmjd2\"" Oct 31 03:43:16.483082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.483052 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl"] Oct 31 03:43:16.560567 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.560518 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259d0ab8-20b4-4c18-9876-962926e794b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl\" (UID: \"259d0ab8-20b4-4c18-9876-962926e794b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 03:43:16.560742 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.560643 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qjfh\" (UniqueName: \"kubernetes.io/projected/259d0ab8-20b4-4c18-9876-962926e794b9-kube-api-access-8qjfh\") pod \"payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl\" (UID: \"259d0ab8-20b4-4c18-9876-962926e794b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 03:43:16.561360 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.561338 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259d0ab8-20b4-4c18-9876-962926e794b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl\" (UID: \"259d0ab8-20b4-4c18-9876-962926e794b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 03:43:16.618335 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.618303 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qjfh\" (UniqueName: \"kubernetes.io/projected/259d0ab8-20b4-4c18-9876-962926e794b9-kube-api-access-8qjfh\") pod \"payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl\" (UID: \"259d0ab8-20b4-4c18-9876-962926e794b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 03:43:16.695088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:16.695052 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 03:43:17.070467 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:17.070435 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl"] Oct 31 03:43:17.071517 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:43:17.071492 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259d0ab8_20b4_4c18_9876_962926e794b9.slice/crio-ef5dd8e2c1fd6926704336b8252b537ce98edd2e9d296bac0abe89a536449647 WatchSource:0}: Error finding container ef5dd8e2c1fd6926704336b8252b537ce98edd2e9d296bac0abe89a536449647: Status 404 returned error can't find the container with id ef5dd8e2c1fd6926704336b8252b537ce98edd2e9d296bac0abe89a536449647 Oct 31 03:43:17.131976 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:17.131945 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" event={"ID":"259d0ab8-20b4-4c18-9876-962926e794b9","Type":"ContainerStarted","Data":"ef5dd8e2c1fd6926704336b8252b537ce98edd2e9d296bac0abe89a536449647"} Oct 31 03:43:18.139826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:18.139779 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" event={"ID":"259d0ab8-20b4-4c18-9876-962926e794b9","Type":"ContainerStarted","Data":"1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8"} Oct 31 03:43:18.140387 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:18.139888 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 03:43:18.249895 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:18.249814 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" podStartSLOduration=2.249799226 podStartE2EDuration="2.249799226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:18.248374367 +0000 UTC m=+895.248999596" watchObservedRunningTime="2026-10-31 03:43:18.249799226 +0000 UTC m=+895.250424454" Oct 31 03:43:18.864905 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:18.864870 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck"] Oct 31 03:43:18.868998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:18.868974 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 03:43:18.962876 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:18.962844 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-68aa3e-dockercfg-z5d66\"" Oct 31 03:43:18.966518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:18.966487 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-68aa3e\"" Oct 31 03:43:18.969492 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:18.969464 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck"] Oct 31 03:43:18.991339 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:18.991301 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7ff47-3bdc-4426-b49a-f36bb921d757-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck\" (UID: \"63c7ff47-3bdc-4426-b49a-f36bb921d757\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 03:43:18.991543 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:18.991459 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qvl\" (UniqueName: \"kubernetes.io/projected/63c7ff47-3bdc-4426-b49a-f36bb921d757-kube-api-access-t2qvl\") pod \"payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck\" (UID: \"63c7ff47-3bdc-4426-b49a-f36bb921d757\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 03:43:19.092922 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:19.092880 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qvl\" (UniqueName: \"kubernetes.io/projected/63c7ff47-3bdc-4426-b49a-f36bb921d757-kube-api-access-t2qvl\") pod \"payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck\" (UID: \"63c7ff47-3bdc-4426-b49a-f36bb921d757\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 03:43:19.093081 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:19.092983 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7ff47-3bdc-4426-b49a-f36bb921d757-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck\" (UID: \"63c7ff47-3bdc-4426-b49a-f36bb921d757\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 03:43:19.093719 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:19.093695 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7ff47-3bdc-4426-b49a-f36bb921d757-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck\" (UID: \"63c7ff47-3bdc-4426-b49a-f36bb921d757\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 03:43:19.150513 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:19.150476 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qvl\" (UniqueName: \"kubernetes.io/projected/63c7ff47-3bdc-4426-b49a-f36bb921d757-kube-api-access-t2qvl\") pod \"payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck\" (UID: \"63c7ff47-3bdc-4426-b49a-f36bb921d757\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 03:43:19.183265 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:19.183213 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 03:43:19.557535 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:19.557341 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck"] Oct 31 03:43:19.562069 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:43:19.562027 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c7ff47_3bdc_4426_b49a_f36bb921d757.slice/crio-1886dd8085cf519a6f5ed9a7f3c1e5893b768305f13a6ba0b856d845573bebe2 WatchSource:0}: Error finding container 1886dd8085cf519a6f5ed9a7f3c1e5893b768305f13a6ba0b856d845573bebe2: Status 404 returned error can't find the container with id 1886dd8085cf519a6f5ed9a7f3c1e5893b768305f13a6ba0b856d845573bebe2 Oct 31 03:43:20.157842 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:20.157796 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" event={"ID":"63c7ff47-3bdc-4426-b49a-f36bb921d757","Type":"ContainerStarted","Data":"06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205"} Oct 31 03:43:20.158332 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:20.157853 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" event={"ID":"63c7ff47-3bdc-4426-b49a-f36bb921d757","Type":"ContainerStarted","Data":"1886dd8085cf519a6f5ed9a7f3c1e5893b768305f13a6ba0b856d845573bebe2"} Oct 31 03:43:20.158332 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:20.157903 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 03:43:20.276739 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:20.276682 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" podStartSLOduration=2.276665873 podStartE2EDuration="2.276665873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:20.273475344 +0000 UTC m=+897.274100573" watchObservedRunningTime="2026-10-31 03:43:20.276665873 +0000 UTC m=+897.277291101" Oct 31 03:43:21.272036 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.271996 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd"] Oct 31 03:43:21.275880 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.275860 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 03:43:21.324939 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.324900 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ae8e27-dockercfg-cfm8p\"" Oct 31 03:43:21.324939 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.324900 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ae8e27\"" Oct 31 03:43:21.374404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.374368 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd"] Oct 31 03:43:21.419536 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.419504 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x927c\" (UniqueName: \"kubernetes.io/projected/e2466146-ded0-4cbe-98d2-168b8ad5227c-kube-api-access-x927c\") pod \"payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd\" (UID: \"e2466146-ded0-4cbe-98d2-168b8ad5227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 03:43:21.419754 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.419569 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2466146-ded0-4cbe-98d2-168b8ad5227c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd\" (UID: \"e2466146-ded0-4cbe-98d2-168b8ad5227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 03:43:21.520639 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.520598 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x927c\" (UniqueName: \"kubernetes.io/projected/e2466146-ded0-4cbe-98d2-168b8ad5227c-kube-api-access-x927c\") pod \"payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd\" (UID: \"e2466146-ded0-4cbe-98d2-168b8ad5227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 03:43:21.520821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.520688 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2466146-ded0-4cbe-98d2-168b8ad5227c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd\" (UID: \"e2466146-ded0-4cbe-98d2-168b8ad5227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 03:43:21.521392 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.521374 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2466146-ded0-4cbe-98d2-168b8ad5227c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd\" (UID: \"e2466146-ded0-4cbe-98d2-168b8ad5227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 03:43:21.580299 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.580221 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x927c\" (UniqueName: \"kubernetes.io/projected/e2466146-ded0-4cbe-98d2-168b8ad5227c-kube-api-access-x927c\") pod \"payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd\" (UID: \"e2466146-ded0-4cbe-98d2-168b8ad5227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 03:43:21.590305 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.590220 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 03:43:21.980450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:21.980420 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd"] Oct 31 03:43:21.981783 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:43:21.981751 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2466146_ded0_4cbe_98d2_168b8ad5227c.slice/crio-1cde44f769397592e6798a9619723ff0bba3a6c694930830990e9a40983b7eac WatchSource:0}: Error finding container 1cde44f769397592e6798a9619723ff0bba3a6c694930830990e9a40983b7eac: Status 404 returned error can't find the container with id 1cde44f769397592e6798a9619723ff0bba3a6c694930830990e9a40983b7eac Oct 31 03:43:22.177258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:22.177218 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" event={"ID":"e2466146-ded0-4cbe-98d2-168b8ad5227c","Type":"ContainerStarted","Data":"1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764"} Oct 31 03:43:22.177428 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:22.177283 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 03:43:22.177428 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:22.177304 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" event={"ID":"e2466146-ded0-4cbe-98d2-168b8ad5227c","Type":"ContainerStarted","Data":"1cde44f769397592e6798a9619723ff0bba3a6c694930830990e9a40983b7eac"} Oct 31 03:43:22.297311 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:22.297203 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" podStartSLOduration=1.297189695 podStartE2EDuration="1.297189695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:22.2936229 +0000 UTC m=+899.294248129" watchObservedRunningTime="2026-10-31 03:43:22.297189695 +0000 UTC m=+899.297814922" Oct 31 03:43:23.202423 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:23.202391 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:43:23.205401 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:23.205375 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:43:24.116041 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:24.116006 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 03:43:27.041339 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:27.041302 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:43:29.149337 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:29.149308 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 03:43:30.360782 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.360748 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t"] Oct 31 03:43:30.365821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.365800 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 03:43:30.425170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.425083 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-d7b7bc-dockercfg-2mldw\"" Oct 31 03:43:30.429759 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.429680 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-d7b7bc\"" Oct 31 03:43:30.473880 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.473838 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t"] Oct 31 03:43:30.475981 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.475007 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgs47\" (UniqueName: \"kubernetes.io/projected/77082b37-d56f-4c57-aa7d-ed0b60b7b049-kube-api-access-vgs47\") pod \"payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t\" (UID: \"77082b37-d56f-4c57-aa7d-ed0b60b7b049\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 03:43:30.476524 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.476496 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77082b37-d56f-4c57-aa7d-ed0b60b7b049-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t\" (UID: \"77082b37-d56f-4c57-aa7d-ed0b60b7b049\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 03:43:30.507401 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.507284 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5"] Oct 31 03:43:30.511513 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.511488 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 03:43:30.567270 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.567234 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5"] Oct 31 03:43:30.578175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.578139 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77082b37-d56f-4c57-aa7d-ed0b60b7b049-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t\" (UID: \"77082b37-d56f-4c57-aa7d-ed0b60b7b049\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 03:43:30.578355 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.578199 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgs47\" (UniqueName: \"kubernetes.io/projected/77082b37-d56f-4c57-aa7d-ed0b60b7b049-kube-api-access-vgs47\") pod \"payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t\" (UID: \"77082b37-d56f-4c57-aa7d-ed0b60b7b049\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 03:43:30.578837 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.578817 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77082b37-d56f-4c57-aa7d-ed0b60b7b049-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t\" (UID: \"77082b37-d56f-4c57-aa7d-ed0b60b7b049\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 03:43:30.642250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.642216 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgs47\" (UniqueName: \"kubernetes.io/projected/77082b37-d56f-4c57-aa7d-ed0b60b7b049-kube-api-access-vgs47\") pod \"payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t\" (UID: \"77082b37-d56f-4c57-aa7d-ed0b60b7b049\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 03:43:30.679397 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.679361 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 03:43:30.679682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.679658 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrkt\" (UniqueName: \"kubernetes.io/projected/d05ee558-ced6-41d0-9bff-c01c0862dc4e-kube-api-access-gxrkt\") pod \"payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5\" (UID: \"d05ee558-ced6-41d0-9bff-c01c0862dc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 03:43:30.679834 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.679815 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05ee558-ced6-41d0-9bff-c01c0862dc4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5\" (UID: \"d05ee558-ced6-41d0-9bff-c01c0862dc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 03:43:30.780830 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.780789 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrkt\" (UniqueName: \"kubernetes.io/projected/d05ee558-ced6-41d0-9bff-c01c0862dc4e-kube-api-access-gxrkt\") pod \"payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5\" (UID: \"d05ee558-ced6-41d0-9bff-c01c0862dc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 03:43:30.781013 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.780892 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05ee558-ced6-41d0-9bff-c01c0862dc4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5\" (UID: \"d05ee558-ced6-41d0-9bff-c01c0862dc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 03:43:30.781852 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.781825 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05ee558-ced6-41d0-9bff-c01c0862dc4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5\" (UID: \"d05ee558-ced6-41d0-9bff-c01c0862dc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 03:43:30.844938 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.844895 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrkt\" (UniqueName: \"kubernetes.io/projected/d05ee558-ced6-41d0-9bff-c01c0862dc4e-kube-api-access-gxrkt\") pod \"payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5\" (UID: \"d05ee558-ced6-41d0-9bff-c01c0862dc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 03:43:30.851410 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:30.851317 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 03:43:31.083661 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:31.083636 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t"] Oct 31 03:43:31.086183 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:43:31.086157 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77082b37_d56f_4c57_aa7d_ed0b60b7b049.slice/crio-0a84919314d7a58a59859cfbd91626bf8f04338e909d9534af50648c932e2e56 WatchSource:0}: Error finding container 0a84919314d7a58a59859cfbd91626bf8f04338e909d9534af50648c932e2e56: Status 404 returned error can't find the container with id 0a84919314d7a58a59859cfbd91626bf8f04338e909d9534af50648c932e2e56 Oct 31 03:43:31.173424 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:31.173392 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 03:43:31.251581 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:31.251521 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5"] Oct 31 03:43:31.275574 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:31.275535 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" event={"ID":"d05ee558-ced6-41d0-9bff-c01c0862dc4e","Type":"ContainerStarted","Data":"c1805ae3f6f1b9065734e6e22b9e0eae886dfecf59d33097e3a88545d46d7cc4"} Oct 31 03:43:31.277448 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:31.277423 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" event={"ID":"77082b37-d56f-4c57-aa7d-ed0b60b7b049","Type":"ContainerStarted","Data":"789a73bf2cd3dd4f4c5808ce8aee17ad2957d98f89d229322539cbe2c332640e"} Oct 31 03:43:31.277576 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:31.277462 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" event={"ID":"77082b37-d56f-4c57-aa7d-ed0b60b7b049","Type":"ContainerStarted","Data":"0a84919314d7a58a59859cfbd91626bf8f04338e909d9534af50648c932e2e56"} Oct 31 03:43:31.277576 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:31.277490 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 03:43:31.409598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:31.409538 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" podStartSLOduration=1.409523775 podStartE2EDuration="1.409523775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:31.406270236 +0000 UTC m=+908.406895463" watchObservedRunningTime="2026-10-31 03:43:31.409523775 +0000 UTC m=+908.410149002" Oct 31 03:43:32.285627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:32.285588 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" event={"ID":"d05ee558-ced6-41d0-9bff-c01c0862dc4e","Type":"ContainerStarted","Data":"56b5688c45eddd7cbbd8260d000e63f3170c66d854bde4d2f33bbc4067ed55fc"} Oct 31 03:43:32.285913 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:32.285761 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 03:43:32.398714 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:32.398656 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" podStartSLOduration=2.39863752 podStartE2EDuration="2.39863752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:32.395174656 +0000 UTC m=+909.395799885" watchObservedRunningTime="2026-10-31 03:43:32.39863752 +0000 UTC m=+909.399262747" Oct 31 03:43:32.874187 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:43:32.874148 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:43:33.187415 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:33.187381 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 03:43:35.240931 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.240895 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq"] Oct 31 03:43:35.245642 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.245612 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 03:43:35.301066 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.301031 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq"] Oct 31 03:43:35.336423 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.336382 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq\" (UID: \"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 03:43:35.336602 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.336434 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mbg\" (UniqueName: \"kubernetes.io/projected/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-kube-api-access-66mbg\") pod \"payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq\" (UID: \"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 03:43:35.343582 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.343554 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-a64fd2-dockercfg-szslr\"" Oct 31 03:43:35.344269 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.344241 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-a64fd2\"" Oct 31 03:43:35.437257 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.437215 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq\" (UID: \"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 03:43:35.437466 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.437269 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mbg\" (UniqueName: \"kubernetes.io/projected/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-kube-api-access-66mbg\") pod \"payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq\" (UID: \"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 03:43:35.437979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.437952 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq\" (UID: \"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 03:43:35.495583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.495500 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mbg\" (UniqueName: \"kubernetes.io/projected/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-kube-api-access-66mbg\") pod \"payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq\" (UID: \"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 03:43:35.561892 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.561842 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 03:43:35.956069 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:35.956040 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq"] Oct 31 03:43:35.956782 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:43:35.956757 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cd7dd5_9af7_40f3_b01f_96eb4f64aae6.slice/crio-b541b1887b864ffe9508eb798d56b20a6a37593e85e10d3f8d33eb5e2c747472 WatchSource:0}: Error finding container b541b1887b864ffe9508eb798d56b20a6a37593e85e10d3f8d33eb5e2c747472: Status 404 returned error can't find the container with id b541b1887b864ffe9508eb798d56b20a6a37593e85e10d3f8d33eb5e2c747472 Oct 31 03:43:36.315556 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:36.315462 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" event={"ID":"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6","Type":"ContainerStarted","Data":"c809e023f856a11cecbb5885eaa9015a97e117d06e93bd20fa26e94ad36541c8"} Oct 31 03:43:36.315556 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:36.315520 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" event={"ID":"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6","Type":"ContainerStarted","Data":"b541b1887b864ffe9508eb798d56b20a6a37593e85e10d3f8d33eb5e2c747472"} Oct 31 03:43:36.315988 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:36.315676 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 03:43:36.426393 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:36.426333 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" podStartSLOduration=1.426316985 podStartE2EDuration="1.426316985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:36.424678932 +0000 UTC m=+913.425304160" watchObservedRunningTime="2026-10-31 03:43:36.426316985 +0000 UTC m=+913.426942209" Oct 31 03:43:36.994108 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:36.993478 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:50260->10.133.0.61:15021: read: connection reset by peer" start-of-body= Oct 31 03:43:36.994108 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:36.993545 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:50260->10.133.0.61:15021: read: connection reset by peer" Oct 31 03:43:37.324652 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.324548 4426 generic.go:356] "Generic (PLEG): container finished" podID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerID="ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675" exitCode=0 Oct 31 03:43:37.324652 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.324630 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" event={"ID":"85150ab8-f8d1-40d2-b540-0f65db33199b","Type":"ContainerDied","Data":"ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675"} Oct 31 03:43:37.325178 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.325159 4426 scope.go:122] "RemoveContainer" containerID="ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675" Oct 31 03:43:37.326574 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.326544 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:37.326658 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.326620 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:37.326658 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.326650 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:37.376496 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.376464 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:43:37.589899 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.589792 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8"] Oct 31 03:43:37.593671 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.593638 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 03:43:37.658144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.653524 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8"] Oct 31 03:43:37.658144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.653951 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-e5b8f9-dockercfg-7zhbz\"" Oct 31 03:43:37.658144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.654059 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-e5b8f9\"" Oct 31 03:43:37.672170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.671765 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64gfv\" (UniqueName: \"kubernetes.io/projected/07b122da-b857-45bb-979d-696f42767f4c-kube-api-access-64gfv\") pod \"payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8\" (UID: \"07b122da-b857-45bb-979d-696f42767f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 03:43:37.672170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.671837 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b122da-b857-45bb-979d-696f42767f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8\" (UID: \"07b122da-b857-45bb-979d-696f42767f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 03:43:37.778066 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.778025 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64gfv\" (UniqueName: \"kubernetes.io/projected/07b122da-b857-45bb-979d-696f42767f4c-kube-api-access-64gfv\") pod \"payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8\" (UID: \"07b122da-b857-45bb-979d-696f42767f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 03:43:37.778290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.778099 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b122da-b857-45bb-979d-696f42767f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8\" (UID: \"07b122da-b857-45bb-979d-696f42767f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 03:43:37.778973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.778948 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b122da-b857-45bb-979d-696f42767f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8\" (UID: \"07b122da-b857-45bb-979d-696f42767f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 03:43:37.852964 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.852880 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64gfv\" (UniqueName: \"kubernetes.io/projected/07b122da-b857-45bb-979d-696f42767f4c-kube-api-access-64gfv\") pod \"payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8\" (UID: \"07b122da-b857-45bb-979d-696f42767f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 03:43:37.908281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:37.908242 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 03:43:38.381191 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:38.379574 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8"] Oct 31 03:43:38.390099 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:38.389993 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" event={"ID":"85150ab8-f8d1-40d2-b540-0f65db33199b","Type":"ContainerStarted","Data":"3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3"} Oct 31 03:43:38.391022 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:43:38.390994 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b122da_b857_45bb_979d_696f42767f4c.slice/crio-10d8fdf0ba5585729c30ae359071c1d3c4ea44583d8c963de3daadb65eba8ca8 WatchSource:0}: Error finding container 10d8fdf0ba5585729c30ae359071c1d3c4ea44583d8c963de3daadb65eba8ca8: Status 404 returned error can't find the container with id 10d8fdf0ba5585729c30ae359071c1d3c4ea44583d8c963de3daadb65eba8ca8 Oct 31 03:43:38.459648 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:38.459558 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:38.460316 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:38.459917 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:43:38.460316 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:38.459975 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 03:43:39.400103 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:39.400070 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" event={"ID":"07b122da-b857-45bb-979d-696f42767f4c","Type":"ContainerStarted","Data":"ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597"} Oct 31 03:43:39.400522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:39.400117 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" event={"ID":"07b122da-b857-45bb-979d-696f42767f4c","Type":"ContainerStarted","Data":"10d8fdf0ba5585729c30ae359071c1d3c4ea44583d8c963de3daadb65eba8ca8"} Oct 31 03:43:39.400522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:39.400203 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 03:43:39.459149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:39.458558 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:43:39.459149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:39.458630 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 03:43:39.513689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:39.513637 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" podStartSLOduration=2.513622559 podStartE2EDuration="2.513622559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:39.511785805 +0000 UTC m=+916.512411032" watchObservedRunningTime="2026-10-31 03:43:39.513622559 +0000 UTC m=+916.514247839" Oct 31 03:43:40.459578 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:40.459537 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:43:40.460712 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:40.460677 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 03:43:41.460148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:41.459255 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:43:41.460148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:41.459338 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 03:43:42.289025 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:42.288995 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 03:43:42.469558 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:42.469516 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:43:42.470186 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:42.470148 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 03:43:43.296252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:43.296222 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 03:43:43.462154 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:43.459540 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:43:43.462154 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:43.459609 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 03:43:44.459346 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:44.459308 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:44.459751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:44.459659 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:44.460743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:44.460720 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 03:43:47.327881 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:47.327854 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 03:43:47.503523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:47.503490 4426 generic.go:356] "Generic (PLEG): container finished" podID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerID="a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760" exitCode=0 Oct 31 03:43:47.503710 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:47.503558 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" event={"ID":"f8f0fec9-03b7-4707-8d26-f40f5d1d312c","Type":"ContainerDied","Data":"a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760"} Oct 31 03:43:47.503957 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:47.503940 4426 scope.go:122] "RemoveContainer" containerID="a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760" Oct 31 03:43:47.505465 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:47.505437 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:47.505572 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:47.505513 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:47.505572 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:47.505566 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:43:48.512082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:48.512049 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" event={"ID":"f8f0fec9-03b7-4707-8d26-f40f5d1d312c","Type":"ContainerStarted","Data":"3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef"} Oct 31 03:43:48.797901 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:48.797814 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:48.799065 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:48.799042 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:43:48.799163 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:48.799091 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 03:43:49.797860 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:49.797825 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:43:49.798340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:49.797879 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 03:43:50.797860 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:50.797820 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:43:50.798376 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:50.797881 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 03:43:51.064563 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:51.064468 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:51.439932 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:51.439900 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 03:43:51.798252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:51.798154 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:43:51.798252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:51.798224 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 03:43:52.797912 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:52.797873 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:43:52.798212 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:52.798183 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 03:43:53.803026 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:53.802990 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:53.804246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:53.804224 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 03:43:54.707038 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:43:54.707000 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:44:02.847642 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:44:02.847598 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:44:06.686820 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:06.686733 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:44:16.767613 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:16.767582 4426 generic.go:356] "Generic (PLEG): container finished" podID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerID="991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889" exitCode=0 Oct 31 03:44:16.768042 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:16.767667 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" event={"ID":"5a8f0271-7150-400d-bd05-da5fa011a1cf","Type":"ContainerDied","Data":"991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889"} Oct 31 03:44:16.768258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:16.768238 4426 scope.go:122] "RemoveContainer" containerID="991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889" Oct 31 03:44:16.769660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:16.769629 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:44:16.769726 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:16.769699 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:44:16.769764 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:16.769728 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:44:17.776384 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:17.776343 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" event={"ID":"5a8f0271-7150-400d-bd05-da5fa011a1cf","Type":"ContainerStarted","Data":"f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7"} Oct 31 03:44:18.135964 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:18.135806 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:44:18.136313 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:18.136285 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:44:18.136518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:18.136495 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 03:44:19.128151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:19.119521 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:44:19.128151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:19.119600 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 03:44:20.123167 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:20.118502 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:44:20.123167 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:20.118582 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 03:44:21.123151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:21.118516 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:44:21.123151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:21.118582 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 03:44:22.070870 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:22.070834 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:44:22.128150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:22.118535 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:44:22.128150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:22.118602 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 03:44:23.116984 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:23.116944 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:44:23.117216 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:23.117010 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 03:44:23.934282 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:23.934247 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:44:24.122503 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:24.122472 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:44:24.123678 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:24.123657 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 03:44:32.849120 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:44:32.849074 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:44:44.356031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:44:44.355993 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:45:00.170464 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.170425 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj"] Oct 31 03:45:00.174842 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.174810 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" Oct 31 03:45:00.178461 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.178430 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6"] Oct 31 03:45:00.182189 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.182166 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld"] Oct 31 03:45:00.182299 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.182265 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" Oct 31 03:45:00.185793 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.185772 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" Oct 31 03:45:00.217254 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.217229 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p"] Oct 31 03:45:00.221559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.221532 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" Oct 31 03:45:00.223922 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.223854 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc"] Oct 31 03:45:00.226955 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.226925 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:45:00.227778 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.227755 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:45:00.230534 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.230509 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh"] Oct 31 03:45:00.230662 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.230623 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" Oct 31 03:45:00.234521 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.234503 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" Oct 31 03:45:00.260373 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.260347 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:45:00.277417 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.277391 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6"] Oct 31 03:45:00.280822 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.280799 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj"] Oct 31 03:45:00.285112 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.285081 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p"] Oct 31 03:45:00.306091 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306059 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jv29\" (UniqueName: \"kubernetes.io/projected/cb3536eb-9c15-43fc-ba82-b7f143484438-kube-api-access-6jv29\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p\" (UID: \"cb3536eb-9c15-43fc-ba82-b7f143484438\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" Oct 31 03:45:00.306504 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306463 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb991d3b-1510-4813-bc64-8c5c61c9614e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc\" (UID: \"cb991d3b-1510-4813-bc64-8c5c61c9614e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" Oct 31 03:45:00.306619 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306525 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3536eb-9c15-43fc-ba82-b7f143484438-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p\" (UID: \"cb3536eb-9c15-43fc-ba82-b7f143484438\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" Oct 31 03:45:00.306619 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306574 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7358276-038a-487d-8e9a-7e03da870287-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6\" (UID: \"b7358276-038a-487d-8e9a-7e03da870287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" Oct 31 03:45:00.306725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306649 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh\" (UID: \"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" Oct 31 03:45:00.306725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306710 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcdkw\" (UniqueName: \"kubernetes.io/projected/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-kube-api-access-rcdkw\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh\" (UID: \"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" Oct 31 03:45:00.306826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306735 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxrt\" (UniqueName: \"kubernetes.io/projected/b7358276-038a-487d-8e9a-7e03da870287-kube-api-access-plxrt\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6\" (UID: \"b7358276-038a-487d-8e9a-7e03da870287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" Oct 31 03:45:00.306826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306765 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240f9af4-2366-4fb8-b203-045e73da25ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj\" (UID: \"240f9af4-2366-4fb8-b203-045e73da25ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" Oct 31 03:45:00.306826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306792 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szk6\" (UniqueName: \"kubernetes.io/projected/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-kube-api-access-7szk6\") pod \"maas-api-key-cleanup-29890305-n84ld\" (UID: \"18e4b978-f8d2-4d70-8ac5-3777558e6ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" Oct 31 03:45:00.306826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306812 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-n84ld\" (UID: \"18e4b978-f8d2-4d70-8ac5-3777558e6ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" Oct 31 03:45:00.307042 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306842 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xght4\" (UniqueName: \"kubernetes.io/projected/cb991d3b-1510-4813-bc64-8c5c61c9614e-kube-api-access-xght4\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc\" (UID: \"cb991d3b-1510-4813-bc64-8c5c61c9614e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" Oct 31 03:45:00.307042 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.306876 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpc42\" (UniqueName: \"kubernetes.io/projected/240f9af4-2366-4fb8-b203-045e73da25ca-kube-api-access-hpc42\") pod \"maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj\" (UID: \"240f9af4-2366-4fb8-b203-045e73da25ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" Oct 31 03:45:00.318398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.318362 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc"] Oct 31 03:45:00.340339 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.340269 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld"] Oct 31 03:45:00.345346 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.345311 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh"] Oct 31 03:45:00.408006 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.407962 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcdkw\" (UniqueName: \"kubernetes.io/projected/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-kube-api-access-rcdkw\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh\" (UID: \"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" Oct 31 03:45:00.408006 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408007 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxrt\" (UniqueName: \"kubernetes.io/projected/b7358276-038a-487d-8e9a-7e03da870287-kube-api-access-plxrt\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6\" (UID: \"b7358276-038a-487d-8e9a-7e03da870287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" Oct 31 03:45:00.408354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408045 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240f9af4-2366-4fb8-b203-045e73da25ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj\" (UID: \"240f9af4-2366-4fb8-b203-045e73da25ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" Oct 31 03:45:00.408354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408071 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szk6\" (UniqueName: \"kubernetes.io/projected/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-kube-api-access-7szk6\") pod \"maas-api-key-cleanup-29890305-n84ld\" (UID: \"18e4b978-f8d2-4d70-8ac5-3777558e6ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" Oct 31 03:45:00.408354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408095 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-n84ld\" (UID: \"18e4b978-f8d2-4d70-8ac5-3777558e6ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" Oct 31 03:45:00.408354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408183 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xght4\" (UniqueName: \"kubernetes.io/projected/cb991d3b-1510-4813-bc64-8c5c61c9614e-kube-api-access-xght4\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc\" (UID: \"cb991d3b-1510-4813-bc64-8c5c61c9614e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" Oct 31 03:45:00.408354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408226 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpc42\" (UniqueName: \"kubernetes.io/projected/240f9af4-2366-4fb8-b203-045e73da25ca-kube-api-access-hpc42\") pod \"maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj\" (UID: \"240f9af4-2366-4fb8-b203-045e73da25ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" Oct 31 03:45:00.408354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408275 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jv29\" (UniqueName: \"kubernetes.io/projected/cb3536eb-9c15-43fc-ba82-b7f143484438-kube-api-access-6jv29\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p\" (UID: \"cb3536eb-9c15-43fc-ba82-b7f143484438\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" Oct 31 03:45:00.408354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408337 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb991d3b-1510-4813-bc64-8c5c61c9614e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc\" (UID: \"cb991d3b-1510-4813-bc64-8c5c61c9614e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" Oct 31 03:45:00.408698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408375 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3536eb-9c15-43fc-ba82-b7f143484438-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p\" (UID: \"cb3536eb-9c15-43fc-ba82-b7f143484438\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" Oct 31 03:45:00.408698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408429 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7358276-038a-487d-8e9a-7e03da870287-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6\" (UID: \"b7358276-038a-487d-8e9a-7e03da870287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" Oct 31 03:45:00.408698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408484 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh\" (UID: \"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" Oct 31 03:45:00.408861 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.408839 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-n84ld\" (UID: \"18e4b978-f8d2-4d70-8ac5-3777558e6ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" Oct 31 03:45:00.409054 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.409027 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240f9af4-2366-4fb8-b203-045e73da25ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj\" (UID: \"240f9af4-2366-4fb8-b203-045e73da25ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" Oct 31 03:45:00.409165 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.409150 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb991d3b-1510-4813-bc64-8c5c61c9614e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc\" (UID: \"cb991d3b-1510-4813-bc64-8c5c61c9614e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" Oct 31 03:45:00.409239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.409207 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3536eb-9c15-43fc-ba82-b7f143484438-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p\" (UID: \"cb3536eb-9c15-43fc-ba82-b7f143484438\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" Oct 31 03:45:00.409297 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.409207 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7358276-038a-487d-8e9a-7e03da870287-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6\" (UID: \"b7358276-038a-487d-8e9a-7e03da870287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" Oct 31 03:45:00.409297 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.409267 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh\" (UID: \"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" Oct 31 03:45:00.466370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.466287 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpc42\" (UniqueName: \"kubernetes.io/projected/240f9af4-2366-4fb8-b203-045e73da25ca-kube-api-access-hpc42\") pod \"maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj\" (UID: \"240f9af4-2366-4fb8-b203-045e73da25ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" Oct 31 03:45:00.466504 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.466444 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxrt\" (UniqueName: \"kubernetes.io/projected/b7358276-038a-487d-8e9a-7e03da870287-kube-api-access-plxrt\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6\" (UID: \"b7358276-038a-487d-8e9a-7e03da870287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" Oct 31 03:45:00.466775 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.466751 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xght4\" (UniqueName: \"kubernetes.io/projected/cb991d3b-1510-4813-bc64-8c5c61c9614e-kube-api-access-xght4\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc\" (UID: \"cb991d3b-1510-4813-bc64-8c5c61c9614e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" Oct 31 03:45:00.466825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.466784 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcdkw\" (UniqueName: \"kubernetes.io/projected/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-kube-api-access-rcdkw\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh\" (UID: \"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" Oct 31 03:45:00.466825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.466803 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jv29\" (UniqueName: \"kubernetes.io/projected/cb3536eb-9c15-43fc-ba82-b7f143484438-kube-api-access-6jv29\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p\" (UID: \"cb3536eb-9c15-43fc-ba82-b7f143484438\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" Oct 31 03:45:00.466825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.466787 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szk6\" (UniqueName: \"kubernetes.io/projected/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-kube-api-access-7szk6\") pod \"maas-api-key-cleanup-29890305-n84ld\" (UID: \"18e4b978-f8d2-4d70-8ac5-3777558e6ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" Oct 31 03:45:00.488521 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.488492 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" Oct 31 03:45:00.498794 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.498772 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" Oct 31 03:45:00.504090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.504062 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" Oct 31 03:45:00.539696 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.539479 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" Oct 31 03:45:00.546976 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.546715 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" Oct 31 03:45:00.554005 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.553736 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" Oct 31 03:45:00.866694 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.866664 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj"] Oct 31 03:45:00.867495 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:45:00.867470 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240f9af4_2366_4fb8_b203_045e73da25ca.slice/crio-b3ebbcd74227c554b49d818939fb840ba154b5d8535095f1e054dad572ac2176 WatchSource:0}: Error finding container b3ebbcd74227c554b49d818939fb840ba154b5d8535095f1e054dad572ac2176: Status 404 returned error can't find the container with id b3ebbcd74227c554b49d818939fb840ba154b5d8535095f1e054dad572ac2176 Oct 31 03:45:00.869554 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.869534 4426 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:45:00.913424 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.913329 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6"] Oct 31 03:45:00.913760 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:45:00.913729 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7358276_038a_487d_8e9a_7e03da870287.slice/crio-584f5fbc1d4f718fb9e6cba8ee047feaf341c93408a81200751d09311e041bc7 WatchSource:0}: Error finding container 584f5fbc1d4f718fb9e6cba8ee047feaf341c93408a81200751d09311e041bc7: Status 404 returned error can't find the container with id 584f5fbc1d4f718fb9e6cba8ee047feaf341c93408a81200751d09311e041bc7 Oct 31 03:45:00.916003 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:45:00.915971 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e4b978_f8d2_4d70_8ac5_3777558e6ac3.slice/crio-ecf7ec54d7a8402747fc91253da4cb0948537eab75317b245f9be7f7bd9ef3db WatchSource:0}: Error finding container ecf7ec54d7a8402747fc91253da4cb0948537eab75317b245f9be7f7bd9ef3db: Status 404 returned error can't find the container with id ecf7ec54d7a8402747fc91253da4cb0948537eab75317b245f9be7f7bd9ef3db Oct 31 03:45:00.917637 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.917615 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld"] Oct 31 03:45:00.988943 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.988904 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc"] Oct 31 03:45:00.989990 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:45:00.989963 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb991d3b_1510_4813_bc64_8c5c61c9614e.slice/crio-4df009476071191406b963e84536da99de95346f1e6d74233a715e7b05f4f2fe WatchSource:0}: Error finding container 4df009476071191406b963e84536da99de95346f1e6d74233a715e7b05f4f2fe: Status 404 returned error can't find the container with id 4df009476071191406b963e84536da99de95346f1e6d74233a715e7b05f4f2fe Oct 31 03:45:00.992170 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:45:00.992146 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf7bd53_33db_44bd_89cc_d60bbe65c1c3.slice/crio-15fd5fe630cf26f425b4e46a022db710800dab0d71290923531933388b95fb7b WatchSource:0}: Error finding container 15fd5fe630cf26f425b4e46a022db710800dab0d71290923531933388b95fb7b: Status 404 returned error can't find the container with id 15fd5fe630cf26f425b4e46a022db710800dab0d71290923531933388b95fb7b Oct 31 03:45:00.992778 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.992756 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh"] Oct 31 03:45:00.998634 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:00.998607 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p"] Oct 31 03:45:00.998835 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:45:00.998812 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3536eb_9c15_43fc_ba82_b7f143484438.slice/crio-8c808e401ddc946d6e0d0c1b753e143f423c032b0b85f56c4f13d64e05791246 WatchSource:0}: Error finding container 8c808e401ddc946d6e0d0c1b753e143f423c032b0b85f56c4f13d64e05791246: Status 404 returned error can't find the container with id 8c808e401ddc946d6e0d0c1b753e143f423c032b0b85f56c4f13d64e05791246 Oct 31 03:45:01.095064 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:01.095026 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" event={"ID":"cb3536eb-9c15-43fc-ba82-b7f143484438","Type":"ContainerStarted","Data":"8c808e401ddc946d6e0d0c1b753e143f423c032b0b85f56c4f13d64e05791246"} Oct 31 03:45:01.096397 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:01.096363 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" event={"ID":"18e4b978-f8d2-4d70-8ac5-3777558e6ac3","Type":"ContainerStarted","Data":"ecf7ec54d7a8402747fc91253da4cb0948537eab75317b245f9be7f7bd9ef3db"} Oct 31 03:45:01.097732 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:01.097701 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" event={"ID":"cb991d3b-1510-4813-bc64-8c5c61c9614e","Type":"ContainerStarted","Data":"4df009476071191406b963e84536da99de95346f1e6d74233a715e7b05f4f2fe"} Oct 31 03:45:01.098950 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:01.098929 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" event={"ID":"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3","Type":"ContainerStarted","Data":"15fd5fe630cf26f425b4e46a022db710800dab0d71290923531933388b95fb7b"} Oct 31 03:45:01.100166 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:01.100144 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" event={"ID":"240f9af4-2366-4fb8-b203-045e73da25ca","Type":"ContainerStarted","Data":"b3ebbcd74227c554b49d818939fb840ba154b5d8535095f1e054dad572ac2176"} Oct 31 03:45:01.101474 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:01.101451 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" event={"ID":"b7358276-038a-487d-8e9a-7e03da870287","Type":"ContainerStarted","Data":"584f5fbc1d4f718fb9e6cba8ee047feaf341c93408a81200751d09311e041bc7"} Oct 31 03:45:02.846841 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:02.846800 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:45:05.365787 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:05.365749 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:45:08.200301 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:08.200255 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" event={"ID":"18e4b978-f8d2-4d70-8ac5-3777558e6ac3","Type":"ContainerStarted","Data":"28a0925bf7cdc7fb1ead88dd1ae6d9153b3ecb3604b66cf87a4cb6f5dce79e7f"} Oct 31 03:45:08.201989 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:08.201955 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" event={"ID":"cb991d3b-1510-4813-bc64-8c5c61c9614e","Type":"ContainerStarted","Data":"cbf963a47027a02a197b4ed911710c4f7be74f883e727b71e7c34745797802be"} Oct 31 03:45:08.203531 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:08.203511 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" event={"ID":"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3","Type":"ContainerStarted","Data":"8e7fa6c89178c4d622c85d15eba05df1cc66469e3a8124fa4e3aacb842d17890"} Oct 31 03:45:08.310544 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:08.310485 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" podStartSLOduration=1.637099193 podStartE2EDuration="8.310471256s" totalImagesPullingTime="6.673372063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:08.307797809 +0000 UTC m=+1005.308423037" watchObservedRunningTime="2026-10-31 03:45:08.310471256 +0000 UTC m=+1005.311096483" Oct 31 03:45:08.417581 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:08.417201 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" podStartSLOduration=1.668796237 podStartE2EDuration="8.41718019s" totalImagesPullingTime="6.748383953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:08.415148879 +0000 UTC m=+1005.415774108" watchObservedRunningTime="2026-10-31 03:45:08.41718019 +0000 UTC m=+1005.417805419" Oct 31 03:45:08.522896 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:08.522844 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" podStartSLOduration=1.712846506 podStartE2EDuration="8.522828001s" totalImagesPullingTime="6.809981495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:08.522451136 +0000 UTC m=+1005.523076368" watchObservedRunningTime="2026-10-31 03:45:08.522828001 +0000 UTC m=+1005.523453228" Oct 31 03:45:09.211679 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.211643 4426 generic.go:356] "Generic (PLEG): container finished" podID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerID="2e9bd3dfd44a69247d24af94d0c80f509a7fa760e40aff73bec498e7a235e87d" exitCode=60 Oct 31 03:45:09.212118 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.211738 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" event={"ID":"cb3536eb-9c15-43fc-ba82-b7f143484438","Type":"ContainerDied","Data":"2e9bd3dfd44a69247d24af94d0c80f509a7fa760e40aff73bec498e7a235e87d"} Oct 31 03:45:09.212118 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.211992 4426 scope.go:122] "RemoveContainer" containerID="2e9bd3dfd44a69247d24af94d0c80f509a7fa760e40aff73bec498e7a235e87d" Oct 31 03:45:09.213400 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.213379 4426 generic.go:356] "Generic (PLEG): container finished" podID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerID="28a0925bf7cdc7fb1ead88dd1ae6d9153b3ecb3604b66cf87a4cb6f5dce79e7f" exitCode=60 Oct 31 03:45:09.213514 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.213463 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" event={"ID":"18e4b978-f8d2-4d70-8ac5-3777558e6ac3","Type":"ContainerDied","Data":"28a0925bf7cdc7fb1ead88dd1ae6d9153b3ecb3604b66cf87a4cb6f5dce79e7f"} Oct 31 03:45:09.213680 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.213660 4426 scope.go:122] "RemoveContainer" containerID="28a0925bf7cdc7fb1ead88dd1ae6d9153b3ecb3604b66cf87a4cb6f5dce79e7f" Oct 31 03:45:09.215202 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.215175 4426 generic.go:356] "Generic (PLEG): container finished" podID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerID="cbf963a47027a02a197b4ed911710c4f7be74f883e727b71e7c34745797802be" exitCode=60 Oct 31 03:45:09.215323 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.215257 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" event={"ID":"cb991d3b-1510-4813-bc64-8c5c61c9614e","Type":"ContainerDied","Data":"cbf963a47027a02a197b4ed911710c4f7be74f883e727b71e7c34745797802be"} Oct 31 03:45:09.215517 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.215455 4426 scope.go:122] "RemoveContainer" containerID="cbf963a47027a02a197b4ed911710c4f7be74f883e727b71e7c34745797802be" Oct 31 03:45:09.217270 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.217253 4426 generic.go:356] "Generic (PLEG): container finished" podID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerID="8e7fa6c89178c4d622c85d15eba05df1cc66469e3a8124fa4e3aacb842d17890" exitCode=60 Oct 31 03:45:09.217369 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.217352 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" event={"ID":"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3","Type":"ContainerDied","Data":"8e7fa6c89178c4d622c85d15eba05df1cc66469e3a8124fa4e3aacb842d17890"} Oct 31 03:45:09.217508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.217491 4426 scope.go:122] "RemoveContainer" containerID="8e7fa6c89178c4d622c85d15eba05df1cc66469e3a8124fa4e3aacb842d17890" Oct 31 03:45:09.219113 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.219093 4426 generic.go:356] "Generic (PLEG): container finished" podID="240f9af4-2366-4fb8-b203-045e73da25ca" containerID="938f21239d6e83ba5a73915de28d3ae09cdbcb61f9761a7666a7f44c68443274" exitCode=60 Oct 31 03:45:09.219210 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.219156 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" event={"ID":"240f9af4-2366-4fb8-b203-045e73da25ca","Type":"ContainerDied","Data":"938f21239d6e83ba5a73915de28d3ae09cdbcb61f9761a7666a7f44c68443274"} Oct 31 03:45:09.219391 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.219375 4426 scope.go:122] "RemoveContainer" containerID="938f21239d6e83ba5a73915de28d3ae09cdbcb61f9761a7666a7f44c68443274" Oct 31 03:45:09.220892 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.220872 4426 generic.go:356] "Generic (PLEG): container finished" podID="b7358276-038a-487d-8e9a-7e03da870287" containerID="084720ff9e0b5ae09b79f796d42cf754d8d63bad9a62eb451dad69b70ff80dd2" exitCode=60 Oct 31 03:45:09.220967 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.220953 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" event={"ID":"b7358276-038a-487d-8e9a-7e03da870287","Type":"ContainerDied","Data":"084720ff9e0b5ae09b79f796d42cf754d8d63bad9a62eb451dad69b70ff80dd2"} Oct 31 03:45:09.221093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:09.221080 4426 scope.go:122] "RemoveContainer" containerID="084720ff9e0b5ae09b79f796d42cf754d8d63bad9a62eb451dad69b70ff80dd2" Oct 31 03:45:10.237534 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.237499 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" event={"ID":"b7358276-038a-487d-8e9a-7e03da870287","Type":"ContainerStarted","Data":"bf158989303a59d45415d04733e71bd5bcb561158bdd4bfcfa06bd046f333087"} Oct 31 03:45:10.239500 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.239472 4426 generic.go:356] "Generic (PLEG): container finished" podID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerID="ec9ac63baf9a6bb3efa040a13d8b66503b3d5cccdfd5eb493415a4ee2c58ab09" exitCode=60 Oct 31 03:45:10.239613 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.239558 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" event={"ID":"cb3536eb-9c15-43fc-ba82-b7f143484438","Type":"ContainerDied","Data":"ec9ac63baf9a6bb3efa040a13d8b66503b3d5cccdfd5eb493415a4ee2c58ab09"} Oct 31 03:45:10.239687 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.239626 4426 scope.go:122] "RemoveContainer" containerID="2e9bd3dfd44a69247d24af94d0c80f509a7fa760e40aff73bec498e7a235e87d" Oct 31 03:45:10.239802 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.239777 4426 scope.go:122] "RemoveContainer" containerID="ec9ac63baf9a6bb3efa040a13d8b66503b3d5cccdfd5eb493415a4ee2c58ab09" Oct 31 03:45:10.240045 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:10.240022 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p_odh-ai-gateway-infra(cb3536eb-9c15-43fc-ba82-b7f143484438)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" Oct 31 03:45:10.242473 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.242451 4426 generic.go:356] "Generic (PLEG): container finished" podID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerID="d820c432c6863f337c5e4c15be8674a907c9d8d28279559cf631472cdf66d77f" exitCode=60 Oct 31 03:45:10.242574 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.242537 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" event={"ID":"18e4b978-f8d2-4d70-8ac5-3777558e6ac3","Type":"ContainerDied","Data":"d820c432c6863f337c5e4c15be8674a907c9d8d28279559cf631472cdf66d77f"} Oct 31 03:45:10.242765 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.242748 4426 scope.go:122] "RemoveContainer" containerID="d820c432c6863f337c5e4c15be8674a907c9d8d28279559cf631472cdf66d77f" Oct 31 03:45:10.243042 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:10.242983 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890305-n84ld_odh-ai-gateway-infra(18e4b978-f8d2-4d70-8ac5-3777558e6ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" Oct 31 03:45:10.244838 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.244810 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" event={"ID":"cb991d3b-1510-4813-bc64-8c5c61c9614e","Type":"ContainerStarted","Data":"327361a701a21c8d7487ebb77f0577833ace8fbe2351f6e00e83d1b9e556572d"} Oct 31 03:45:10.245079 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.245061 4426 scope.go:122] "RemoveContainer" containerID="327361a701a21c8d7487ebb77f0577833ace8fbe2351f6e00e83d1b9e556572d" Oct 31 03:45:10.245423 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:10.245398 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc_odh-ai-gateway-infra(cb991d3b-1510-4813-bc64-8c5c61c9614e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" Oct 31 03:45:10.246993 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.246970 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" event={"ID":"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3","Type":"ContainerStarted","Data":"09d163440b1cd92e6e756d72b4c8728dbe885a44395ae22dbde51f020587a2ae"} Oct 31 03:45:10.248991 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.248970 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" event={"ID":"240f9af4-2366-4fb8-b203-045e73da25ca","Type":"ContainerStarted","Data":"078b8d146dc49403fda91535bbf47e4745c609427cbdca70c8a75492b0ac07a4"} Oct 31 03:45:10.314483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.314450 4426 scope.go:122] "RemoveContainer" containerID="28a0925bf7cdc7fb1ead88dd1ae6d9153b3ecb3604b66cf87a4cb6f5dce79e7f" Oct 31 03:45:10.347025 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.346975 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" podStartSLOduration=3.359410189 podStartE2EDuration="10.346960942s" totalImagesPullingTime="6.987550753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:10.345478726 +0000 UTC m=+1007.346103954" watchObservedRunningTime="2026-10-31 03:45:10.346960942 +0000 UTC m=+1007.347586169" Oct 31 03:45:10.456406 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:10.456339 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" podStartSLOduration=3.438077016 podStartE2EDuration="10.456319334s" totalImagesPullingTime="7.018242318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:10.453505865 +0000 UTC m=+1007.454131093" watchObservedRunningTime="2026-10-31 03:45:10.456319334 +0000 UTC m=+1007.456944565" Oct 31 03:45:11.257748 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.257709 4426 generic.go:356] "Generic (PLEG): container finished" podID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerID="09d163440b1cd92e6e756d72b4c8728dbe885a44395ae22dbde51f020587a2ae" exitCode=60 Oct 31 03:45:11.258243 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.257791 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" event={"ID":"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3","Type":"ContainerDied","Data":"09d163440b1cd92e6e756d72b4c8728dbe885a44395ae22dbde51f020587a2ae"} Oct 31 03:45:11.258243 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.257851 4426 scope.go:122] "RemoveContainer" containerID="8e7fa6c89178c4d622c85d15eba05df1cc66469e3a8124fa4e3aacb842d17890" Oct 31 03:45:11.258243 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.258111 4426 scope.go:122] "RemoveContainer" containerID="09d163440b1cd92e6e756d72b4c8728dbe885a44395ae22dbde51f020587a2ae" Oct 31 03:45:11.258454 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:11.258425 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh_odh-ai-gateway-infra(8bf7bd53-33db-44bd-89cc-d60bbe65c1c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" Oct 31 03:45:11.259825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.259795 4426 generic.go:356] "Generic (PLEG): container finished" podID="240f9af4-2366-4fb8-b203-045e73da25ca" containerID="078b8d146dc49403fda91535bbf47e4745c609427cbdca70c8a75492b0ac07a4" exitCode=60 Oct 31 03:45:11.259900 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.259862 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" event={"ID":"240f9af4-2366-4fb8-b203-045e73da25ca","Type":"ContainerDied","Data":"078b8d146dc49403fda91535bbf47e4745c609427cbdca70c8a75492b0ac07a4"} Oct 31 03:45:11.260083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.260069 4426 scope.go:122] "RemoveContainer" containerID="078b8d146dc49403fda91535bbf47e4745c609427cbdca70c8a75492b0ac07a4" Oct 31 03:45:11.260342 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:11.260324 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj_odh-ai-gateway-infra(240f9af4-2366-4fb8-b203-045e73da25ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" Oct 31 03:45:11.261859 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.261836 4426 generic.go:356] "Generic (PLEG): container finished" podID="b7358276-038a-487d-8e9a-7e03da870287" containerID="bf158989303a59d45415d04733e71bd5bcb561158bdd4bfcfa06bd046f333087" exitCode=60 Oct 31 03:45:11.261939 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.261920 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" event={"ID":"b7358276-038a-487d-8e9a-7e03da870287","Type":"ContainerDied","Data":"bf158989303a59d45415d04733e71bd5bcb561158bdd4bfcfa06bd046f333087"} Oct 31 03:45:11.262095 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.262081 4426 scope.go:122] "RemoveContainer" containerID="bf158989303a59d45415d04733e71bd5bcb561158bdd4bfcfa06bd046f333087" Oct 31 03:45:11.262322 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:11.262305 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6_odh-ai-gateway-infra(b7358276-038a-487d-8e9a-7e03da870287)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" podUID="b7358276-038a-487d-8e9a-7e03da870287" Oct 31 03:45:11.263720 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.263703 4426 scope.go:122] "RemoveContainer" containerID="ec9ac63baf9a6bb3efa040a13d8b66503b3d5cccdfd5eb493415a4ee2c58ab09" Oct 31 03:45:11.263909 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:11.263887 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p_odh-ai-gateway-infra(cb3536eb-9c15-43fc-ba82-b7f143484438)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" Oct 31 03:45:11.265339 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.265321 4426 scope.go:122] "RemoveContainer" containerID="d820c432c6863f337c5e4c15be8674a907c9d8d28279559cf631472cdf66d77f" Oct 31 03:45:11.265515 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:11.265498 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890305-n84ld_odh-ai-gateway-infra(18e4b978-f8d2-4d70-8ac5-3777558e6ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" Oct 31 03:45:11.267167 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.267111 4426 generic.go:356] "Generic (PLEG): container finished" podID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerID="327361a701a21c8d7487ebb77f0577833ace8fbe2351f6e00e83d1b9e556572d" exitCode=60 Oct 31 03:45:11.267256 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.267162 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" event={"ID":"cb991d3b-1510-4813-bc64-8c5c61c9614e","Type":"ContainerDied","Data":"327361a701a21c8d7487ebb77f0577833ace8fbe2351f6e00e83d1b9e556572d"} Oct 31 03:45:11.267419 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.267402 4426 scope.go:122] "RemoveContainer" containerID="327361a701a21c8d7487ebb77f0577833ace8fbe2351f6e00e83d1b9e556572d" Oct 31 03:45:11.267635 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:11.267614 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc_odh-ai-gateway-infra(cb991d3b-1510-4813-bc64-8c5c61c9614e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" Oct 31 03:45:11.270859 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.270844 4426 scope.go:122] "RemoveContainer" containerID="938f21239d6e83ba5a73915de28d3ae09cdbcb61f9761a7666a7f44c68443274" Oct 31 03:45:11.284400 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.284377 4426 scope.go:122] "RemoveContainer" containerID="084720ff9e0b5ae09b79f796d42cf754d8d63bad9a62eb451dad69b70ff80dd2" Oct 31 03:45:11.297311 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:11.297289 4426 scope.go:122] "RemoveContainer" containerID="cbf963a47027a02a197b4ed911710c4f7be74f883e727b71e7c34745797802be" Oct 31 03:45:12.276086 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:12.276052 4426 scope.go:122] "RemoveContainer" containerID="327361a701a21c8d7487ebb77f0577833ace8fbe2351f6e00e83d1b9e556572d" Oct 31 03:45:12.276532 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:12.276371 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc_odh-ai-gateway-infra(cb991d3b-1510-4813-bc64-8c5c61c9614e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" Oct 31 03:45:12.277677 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:12.277658 4426 scope.go:122] "RemoveContainer" containerID="09d163440b1cd92e6e756d72b4c8728dbe885a44395ae22dbde51f020587a2ae" Oct 31 03:45:12.277897 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:12.277874 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh_odh-ai-gateway-infra(8bf7bd53-33db-44bd-89cc-d60bbe65c1c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" Oct 31 03:45:12.279196 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:12.279179 4426 scope.go:122] "RemoveContainer" containerID="078b8d146dc49403fda91535bbf47e4745c609427cbdca70c8a75492b0ac07a4" Oct 31 03:45:12.279414 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:12.279395 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj_odh-ai-gateway-infra(240f9af4-2366-4fb8-b203-045e73da25ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" Oct 31 03:45:12.280670 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:12.280655 4426 scope.go:122] "RemoveContainer" containerID="bf158989303a59d45415d04733e71bd5bcb561158bdd4bfcfa06bd046f333087" Oct 31 03:45:12.280819 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:12.280804 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6_odh-ai-gateway-infra(b7358276-038a-487d-8e9a-7e03da870287)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" podUID="b7358276-038a-487d-8e9a-7e03da870287" Oct 31 03:45:15.835835 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:15.835798 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl"] Oct 31 03:45:15.841370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:15.841347 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:15.895920 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:15.895885 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl"] Oct 31 03:45:15.938879 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:15.938849 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-68aa3e\"/\"kube-root-ca.crt\"" Oct 31 03:45:15.939183 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:15.938847 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-68aa3e\"/\"openshift-service-ca.crt\"" Oct 31 03:45:15.939183 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:15.938874 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68aa3e\"/\"default-dockercfg-bjffd\"" Oct 31 03:45:15.939183 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:15.938983 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68aa3e\"/\"premium-w4-68aa3e-kserve-self-signed-certs\"" Oct 31 03:45:16.000000 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:15.999959 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tls-certs\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.000471 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.000346 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-home\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.000741 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.000720 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blq4j\" (UniqueName: \"kubernetes.io/projected/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-kube-api-access-blq4j\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.001508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.001484 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-model-cache\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.001724 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.001700 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-dshm\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.002452 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.002420 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tmp-dir\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.103359 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.103265 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tmp-dir\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.103359 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.103328 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tls-certs\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.103359 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.103344 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-home\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.103649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.103363 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blq4j\" (UniqueName: \"kubernetes.io/projected/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-kube-api-access-blq4j\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.103649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.103400 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-model-cache\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.103649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.103573 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-dshm\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.103810 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.103733 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tmp-dir\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.103810 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.103768 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-home\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.103893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.103805 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-model-cache\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.105753 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.105724 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-dshm\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.105888 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.105826 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tls-certs\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.161059 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.161025 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blq4j\" (UniqueName: \"kubernetes.io/projected/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-kube-api-access-blq4j\") pod \"premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.454282 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.454236 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:16.827398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:16.827356 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl"] Oct 31 03:45:16.828106 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:45:16.828069 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b78aebf_ac0b_4b51_a1a4_4d7e11a4e565.slice/crio-e934a7d2a67fc5b1f37d7d149e6272eb578eecfa3ea3cf77df44ec3ee4aae4b6 WatchSource:0}: Error finding container e934a7d2a67fc5b1f37d7d149e6272eb578eecfa3ea3cf77df44ec3ee4aae4b6: Status 404 returned error can't find the container with id e934a7d2a67fc5b1f37d7d149e6272eb578eecfa3ea3cf77df44ec3ee4aae4b6 Oct 31 03:45:17.316169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:17.316108 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" event={"ID":"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565","Type":"ContainerStarted","Data":"2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284"} Oct 31 03:45:17.316574 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:17.316179 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" event={"ID":"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565","Type":"ContainerStarted","Data":"e934a7d2a67fc5b1f37d7d149e6272eb578eecfa3ea3cf77df44ec3ee4aae4b6"} Oct 31 03:45:17.429353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:17.429291 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podStartSLOduration=2.429273757 podStartE2EDuration="2.429273757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:17.426038843 +0000 UTC m=+1014.426664053" watchObservedRunningTime="2026-10-31 03:45:17.429273757 +0000 UTC m=+1014.429898984" Oct 31 03:45:17.454995 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:17.454961 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:21.122762 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:21.122723 4426 scope.go:122] "RemoveContainer" containerID="327361a701a21c8d7487ebb77f0577833ace8fbe2351f6e00e83d1b9e556572d" Oct 31 03:45:22.116810 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:22.116770 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:45:22.121814 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:22.121781 4426 scope.go:122] "RemoveContainer" containerID="ec9ac63baf9a6bb3efa040a13d8b66503b3d5cccdfd5eb493415a4ee2c58ab09" Oct 31 03:45:22.354118 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:22.354028 4426 generic.go:356] "Generic (PLEG): container finished" podID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerID="60d1fb98ce0f640b19520c44ba5238272f9baab988f2f90cde0dca47df218868" exitCode=60 Oct 31 03:45:22.354531 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:22.354167 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" event={"ID":"cb991d3b-1510-4813-bc64-8c5c61c9614e","Type":"ContainerDied","Data":"60d1fb98ce0f640b19520c44ba5238272f9baab988f2f90cde0dca47df218868"} Oct 31 03:45:22.354531 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:22.354226 4426 scope.go:122] "RemoveContainer" containerID="327361a701a21c8d7487ebb77f0577833ace8fbe2351f6e00e83d1b9e556572d" Oct 31 03:45:22.354712 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:22.354694 4426 scope.go:122] "RemoveContainer" containerID="60d1fb98ce0f640b19520c44ba5238272f9baab988f2f90cde0dca47df218868" Oct 31 03:45:22.355044 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:22.355012 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc_odh-ai-gateway-infra(cb991d3b-1510-4813-bc64-8c5c61c9614e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" Oct 31 03:45:23.127600 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.127572 4426 scope.go:122] "RemoveContainer" containerID="d820c432c6863f337c5e4c15be8674a907c9d8d28279559cf631472cdf66d77f" Oct 31 03:45:23.363341 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.363309 4426 generic.go:356] "Generic (PLEG): container finished" podID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerID="53192051372ee28995a105634301b4e1170c52e5986c05b781f706f796e843c5" exitCode=60 Oct 31 03:45:23.363821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.363386 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" event={"ID":"cb3536eb-9c15-43fc-ba82-b7f143484438","Type":"ContainerDied","Data":"53192051372ee28995a105634301b4e1170c52e5986c05b781f706f796e843c5"} Oct 31 03:45:23.363821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.363461 4426 scope.go:122] "RemoveContainer" containerID="ec9ac63baf9a6bb3efa040a13d8b66503b3d5cccdfd5eb493415a4ee2c58ab09" Oct 31 03:45:23.363931 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.363857 4426 scope.go:122] "RemoveContainer" containerID="53192051372ee28995a105634301b4e1170c52e5986c05b781f706f796e843c5" Oct 31 03:45:23.364264 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:23.364232 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p_odh-ai-gateway-infra(cb3536eb-9c15-43fc-ba82-b7f143484438)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" Oct 31 03:45:23.474427 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.474392 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc"] Oct 31 03:45:23.642695 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.642667 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" Oct 31 03:45:23.685924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.685888 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb991d3b-1510-4813-bc64-8c5c61c9614e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb991d3b-1510-4813-bc64-8c5c61c9614e-maas-api-service-ca\") pod \"cb991d3b-1510-4813-bc64-8c5c61c9614e\" (UID: \"cb991d3b-1510-4813-bc64-8c5c61c9614e\") " Oct 31 03:45:23.686105 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.685985 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb991d3b-1510-4813-bc64-8c5c61c9614e-kube-api-access-xght4\" (UniqueName: \"kubernetes.io/projected/cb991d3b-1510-4813-bc64-8c5c61c9614e-kube-api-access-xght4\") pod \"cb991d3b-1510-4813-bc64-8c5c61c9614e\" (UID: \"cb991d3b-1510-4813-bc64-8c5c61c9614e\") " Oct 31 03:45:23.686327 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.686301 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb991d3b-1510-4813-bc64-8c5c61c9614e-maas-api-service-ca" pod "cb991d3b-1510-4813-bc64-8c5c61c9614e" (UID: "cb991d3b-1510-4813-bc64-8c5c61c9614e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:23.688155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.688115 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb991d3b-1510-4813-bc64-8c5c61c9614e-kube-api-access-xght4" pod "cb991d3b-1510-4813-bc64-8c5c61c9614e" (UID: "cb991d3b-1510-4813-bc64-8c5c61c9614e"). InnerVolumeSpecName "kube-api-access-xght4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:23.787302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.787265 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb991d3b-1510-4813-bc64-8c5c61c9614e-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:23.787302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:23.787302 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xght4\" (UniqueName: \"kubernetes.io/projected/cb991d3b-1510-4813-bc64-8c5c61c9614e-kube-api-access-xght4\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:24.048078 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.048043 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d"] Oct 31 03:45:24.048611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.048597 4426 cpu_manager.go:414] "containerMap: removing container" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerName="cleanup" Oct 31 03:45:24.048660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.048613 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerName="cleanup" Oct 31 03:45:24.048660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.048632 4426 cpu_manager.go:414] "containerMap: removing container" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerName="cleanup" Oct 31 03:45:24.048660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.048639 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerName="cleanup" Oct 31 03:45:24.048754 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.048727 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerName="cleanup" Oct 31 03:45:24.048754 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.048739 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerName="cleanup" Oct 31 03:45:24.052068 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.052027 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.090497 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.090452 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-home\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.090699 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.090589 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-dshm\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.090699 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.090635 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42350f0d-272a-48cf-bfbf-9c60ffa55221-tls-certs\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.090699 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.090670 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-tmp-dir\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.090875 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.090742 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-model-cache\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.090875 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.090839 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn225\" (UniqueName: \"kubernetes.io/projected/42350f0d-272a-48cf-bfbf-9c60ffa55221-kube-api-access-wn225\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.104084 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.104042 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e5b8f9\"/\"openshift-service-ca.crt\"" Oct 31 03:45:24.109845 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.109808 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e5b8f9\"/\"kube-root-ca.crt\"" Oct 31 03:45:24.110435 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.110414 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d"] Oct 31 03:45:24.123087 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.123051 4426 scope.go:122] "RemoveContainer" containerID="078b8d146dc49403fda91535bbf47e4745c609427cbdca70c8a75492b0ac07a4" Oct 31 03:45:24.123322 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.123282 4426 scope.go:122] "RemoveContainer" containerID="bf158989303a59d45415d04733e71bd5bcb561158bdd4bfcfa06bd046f333087" Oct 31 03:45:24.150184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.150153 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e5b8f9\"/\"simulator-w5-e5b8f9-kserve-self-signed-certs\"" Oct 31 03:45:24.150184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.150161 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e5b8f9\"/\"default-dockercfg-57f6t\"" Oct 31 03:45:24.200636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.192325 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-home\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.200636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.192502 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-dshm\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.200636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.192597 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42350f0d-272a-48cf-bfbf-9c60ffa55221-tls-certs\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.200636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.192739 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-tmp-dir\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.200636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.192785 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-model-cache\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.200636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.192875 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn225\" (UniqueName: \"kubernetes.io/projected/42350f0d-272a-48cf-bfbf-9c60ffa55221-kube-api-access-wn225\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.200636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.194466 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-home\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.201301 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.201012 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-tmp-dir\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.201301 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.201264 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-model-cache\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.206075 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.206045 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42350f0d-272a-48cf-bfbf-9c60ffa55221-tls-certs\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.209833 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.209796 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-dshm\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.315625 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.315594 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn225\" (UniqueName: \"kubernetes.io/projected/42350f0d-272a-48cf-bfbf-9c60ffa55221-kube-api-access-wn225\") pod \"simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.365705 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.365660 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:24.382652 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.382618 4426 generic.go:356] "Generic (PLEG): container finished" podID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerID="2b3bac3f18c3de240bba51402083eaa0f63d2bf37dab37f8bab7a0a617633f65" exitCode=60 Oct 31 03:45:24.382799 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.382700 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" event={"ID":"18e4b978-f8d2-4d70-8ac5-3777558e6ac3","Type":"ContainerDied","Data":"2b3bac3f18c3de240bba51402083eaa0f63d2bf37dab37f8bab7a0a617633f65"} Oct 31 03:45:24.382799 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.382779 4426 scope.go:122] "RemoveContainer" containerID="d820c432c6863f337c5e4c15be8674a907c9d8d28279559cf631472cdf66d77f" Oct 31 03:45:24.383222 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.383204 4426 scope.go:122] "RemoveContainer" containerID="2b3bac3f18c3de240bba51402083eaa0f63d2bf37dab37f8bab7a0a617633f65" Oct 31 03:45:24.383515 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:24.383488 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890305-n84ld_odh-ai-gateway-infra(18e4b978-f8d2-4d70-8ac5-3777558e6ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" Oct 31 03:45:24.384744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.384726 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" Oct 31 03:45:24.384845 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.384757 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc" event={"ID":"cb991d3b-1510-4813-bc64-8c5c61c9614e","Type":"ContainerDied","Data":"4df009476071191406b963e84536da99de95346f1e6d74233a715e7b05f4f2fe"} Oct 31 03:45:24.399927 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.399894 4426 scope.go:122] "RemoveContainer" containerID="60d1fb98ce0f640b19520c44ba5238272f9baab988f2f90cde0dca47df218868" Oct 31 03:45:24.488649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.488607 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p"] Oct 31 03:45:24.605930 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.605899 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc"] Oct 31 03:45:24.610950 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.610918 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890305-ftzkc"] Oct 31 03:45:24.650589 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.650562 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" Oct 31 03:45:24.697390 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.697280 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3536eb-9c15-43fc-ba82-b7f143484438-kube-api-access-6jv29\" (UniqueName: \"kubernetes.io/projected/cb3536eb-9c15-43fc-ba82-b7f143484438-kube-api-access-6jv29\") pod \"cb3536eb-9c15-43fc-ba82-b7f143484438\" (UID: \"cb3536eb-9c15-43fc-ba82-b7f143484438\") " Oct 31 03:45:24.697390 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.697329 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3536eb-9c15-43fc-ba82-b7f143484438-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3536eb-9c15-43fc-ba82-b7f143484438-maas-api-service-ca\") pod \"cb3536eb-9c15-43fc-ba82-b7f143484438\" (UID: \"cb3536eb-9c15-43fc-ba82-b7f143484438\") " Oct 31 03:45:24.697721 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.697698 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3536eb-9c15-43fc-ba82-b7f143484438-maas-api-service-ca" pod "cb3536eb-9c15-43fc-ba82-b7f143484438" (UID: "cb3536eb-9c15-43fc-ba82-b7f143484438"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:24.697855 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.697836 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3536eb-9c15-43fc-ba82-b7f143484438-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:24.699523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.699497 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3536eb-9c15-43fc-ba82-b7f143484438-kube-api-access-6jv29" pod "cb3536eb-9c15-43fc-ba82-b7f143484438" (UID: "cb3536eb-9c15-43fc-ba82-b7f143484438"). InnerVolumeSpecName "kube-api-access-6jv29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:24.753967 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.753932 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d"] Oct 31 03:45:24.755574 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:45:24.755546 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42350f0d_272a_48cf_bfbf_9c60ffa55221.slice/crio-915f81d00de30edaef70330efa8b6aa274839c33ab238bb6168bbef30be47a9b WatchSource:0}: Error finding container 915f81d00de30edaef70330efa8b6aa274839c33ab238bb6168bbef30be47a9b: Status 404 returned error can't find the container with id 915f81d00de30edaef70330efa8b6aa274839c33ab238bb6168bbef30be47a9b Oct 31 03:45:24.798577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:24.798549 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jv29\" (UniqueName: \"kubernetes.io/projected/cb3536eb-9c15-43fc-ba82-b7f143484438-kube-api-access-6jv29\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:25.121901 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.121873 4426 scope.go:122] "RemoveContainer" containerID="09d163440b1cd92e6e756d72b4c8728dbe885a44395ae22dbde51f020587a2ae" Oct 31 03:45:25.132287 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.132261 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" path="/var/lib/kubelet/pods/cb991d3b-1510-4813-bc64-8c5c61c9614e/volumes" Oct 31 03:45:25.392686 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.392650 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" event={"ID":"42350f0d-272a-48cf-bfbf-9c60ffa55221","Type":"ContainerStarted","Data":"b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8"} Oct 31 03:45:25.393177 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.392712 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" event={"ID":"42350f0d-272a-48cf-bfbf-9c60ffa55221","Type":"ContainerStarted","Data":"915f81d00de30edaef70330efa8b6aa274839c33ab238bb6168bbef30be47a9b"} Oct 31 03:45:25.394981 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.394951 4426 generic.go:356] "Generic (PLEG): container finished" podID="240f9af4-2366-4fb8-b203-045e73da25ca" containerID="94c2a82cbbaba2e692e55188b8dee8d3d2abbc9bcdc602d00a46407a8321395e" exitCode=60 Oct 31 03:45:25.395155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.395001 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" event={"ID":"240f9af4-2366-4fb8-b203-045e73da25ca","Type":"ContainerDied","Data":"94c2a82cbbaba2e692e55188b8dee8d3d2abbc9bcdc602d00a46407a8321395e"} Oct 31 03:45:25.395155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.395062 4426 scope.go:122] "RemoveContainer" containerID="078b8d146dc49403fda91535bbf47e4745c609427cbdca70c8a75492b0ac07a4" Oct 31 03:45:25.395456 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.395428 4426 scope.go:122] "RemoveContainer" containerID="94c2a82cbbaba2e692e55188b8dee8d3d2abbc9bcdc602d00a46407a8321395e" Oct 31 03:45:25.395733 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:25.395706 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj_odh-ai-gateway-infra(240f9af4-2366-4fb8-b203-045e73da25ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" Oct 31 03:45:25.397461 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.397435 4426 generic.go:356] "Generic (PLEG): container finished" podID="b7358276-038a-487d-8e9a-7e03da870287" containerID="c43bf6324cdfe8bada49a31ce632824dd998b88ec2701c57d3db9bdeeab7919c" exitCode=60 Oct 31 03:45:25.397555 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.397499 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" event={"ID":"b7358276-038a-487d-8e9a-7e03da870287","Type":"ContainerDied","Data":"c43bf6324cdfe8bada49a31ce632824dd998b88ec2701c57d3db9bdeeab7919c"} Oct 31 03:45:25.397826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.397809 4426 scope.go:122] "RemoveContainer" containerID="c43bf6324cdfe8bada49a31ce632824dd998b88ec2701c57d3db9bdeeab7919c" Oct 31 03:45:25.398074 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:25.398055 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6_odh-ai-gateway-infra(b7358276-038a-487d-8e9a-7e03da870287)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" podUID="b7358276-038a-487d-8e9a-7e03da870287" Oct 31 03:45:25.399011 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.398990 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" Oct 31 03:45:25.399118 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.399025 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p" event={"ID":"cb3536eb-9c15-43fc-ba82-b7f143484438","Type":"ContainerDied","Data":"8c808e401ddc946d6e0d0c1b753e143f423c032b0b85f56c4f13d64e05791246"} Oct 31 03:45:25.410062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.410042 4426 scope.go:122] "RemoveContainer" containerID="bf158989303a59d45415d04733e71bd5bcb561158bdd4bfcfa06bd046f333087" Oct 31 03:45:25.424236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.424217 4426 scope.go:122] "RemoveContainer" containerID="53192051372ee28995a105634301b4e1170c52e5986c05b781f706f796e843c5" Oct 31 03:45:25.510291 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.510237 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podStartSLOduration=1.510222001 podStartE2EDuration="1.510222001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:25.506270071 +0000 UTC m=+1022.506895320" watchObservedRunningTime="2026-10-31 03:45:25.510222001 +0000 UTC m=+1022.510847230" Oct 31 03:45:25.512239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.512220 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld"] Oct 31 03:45:25.618530 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.618439 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p"] Oct 31 03:45:25.623604 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.623571 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890305-9n52p"] Oct 31 03:45:25.658276 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.658253 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" Oct 31 03:45:25.707767 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.707733 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-maas-api-service-ca\") pod \"18e4b978-f8d2-4d70-8ac5-3777558e6ac3\" (UID: \"18e4b978-f8d2-4d70-8ac5-3777558e6ac3\") " Oct 31 03:45:25.707961 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.707779 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-kube-api-access-7szk6\" (UniqueName: \"kubernetes.io/projected/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-kube-api-access-7szk6\") pod \"18e4b978-f8d2-4d70-8ac5-3777558e6ac3\" (UID: \"18e4b978-f8d2-4d70-8ac5-3777558e6ac3\") " Oct 31 03:45:25.708088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.708063 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-maas-api-service-ca" pod "18e4b978-f8d2-4d70-8ac5-3777558e6ac3" (UID: "18e4b978-f8d2-4d70-8ac5-3777558e6ac3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:25.708199 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.708185 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:25.709817 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.709793 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-kube-api-access-7szk6" pod "18e4b978-f8d2-4d70-8ac5-3777558e6ac3" (UID: "18e4b978-f8d2-4d70-8ac5-3777558e6ac3"). InnerVolumeSpecName "kube-api-access-7szk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:25.809270 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:25.809226 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szk6\" (UniqueName: \"kubernetes.io/projected/18e4b978-f8d2-4d70-8ac5-3777558e6ac3-kube-api-access-7szk6\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:26.366150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.366059 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:26.415362 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.415334 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" Oct 31 03:45:26.415883 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.415371 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld" event={"ID":"18e4b978-f8d2-4d70-8ac5-3777558e6ac3","Type":"ContainerDied","Data":"ecf7ec54d7a8402747fc91253da4cb0948537eab75317b245f9be7f7bd9ef3db"} Oct 31 03:45:26.415883 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.415436 4426 scope.go:122] "RemoveContainer" containerID="2b3bac3f18c3de240bba51402083eaa0f63d2bf37dab37f8bab7a0a617633f65" Oct 31 03:45:26.422308 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.422279 4426 generic.go:356] "Generic (PLEG): container finished" podID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerID="1dc1062c9448cf7fc374f1989d9626a0dab3169f0c7d2f441e047c816a85686b" exitCode=60 Oct 31 03:45:26.422511 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.422327 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" event={"ID":"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3","Type":"ContainerDied","Data":"1dc1062c9448cf7fc374f1989d9626a0dab3169f0c7d2f441e047c816a85686b"} Oct 31 03:45:26.423008 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.422985 4426 scope.go:122] "RemoveContainer" containerID="1dc1062c9448cf7fc374f1989d9626a0dab3169f0c7d2f441e047c816a85686b" Oct 31 03:45:26.423259 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:26.423237 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh_odh-ai-gateway-infra(8bf7bd53-33db-44bd-89cc-d60bbe65c1c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" Oct 31 03:45:26.427368 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.427349 4426 scope.go:122] "RemoveContainer" containerID="09d163440b1cd92e6e756d72b4c8728dbe885a44395ae22dbde51f020587a2ae" Oct 31 03:45:26.454406 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.454376 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:26.468039 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.468015 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:26.572154 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.572103 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld"] Oct 31 03:45:26.576002 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.575975 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-n84ld"] Oct 31 03:45:26.743088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.743055 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6"] Oct 31 03:45:26.854235 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.854197 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj"] Oct 31 03:45:26.901978 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:26.901950 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" Oct 31 03:45:27.019234 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.019196 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7358276-038a-487d-8e9a-7e03da870287-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7358276-038a-487d-8e9a-7e03da870287-maas-api-service-ca\") pod \"b7358276-038a-487d-8e9a-7e03da870287\" (UID: \"b7358276-038a-487d-8e9a-7e03da870287\") " Oct 31 03:45:27.019401 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.019331 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7358276-038a-487d-8e9a-7e03da870287-kube-api-access-plxrt\" (UniqueName: \"kubernetes.io/projected/b7358276-038a-487d-8e9a-7e03da870287-kube-api-access-plxrt\") pod \"b7358276-038a-487d-8e9a-7e03da870287\" (UID: \"b7358276-038a-487d-8e9a-7e03da870287\") " Oct 31 03:45:27.019590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.019555 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7358276-038a-487d-8e9a-7e03da870287-maas-api-service-ca" pod "b7358276-038a-487d-8e9a-7e03da870287" (UID: "b7358276-038a-487d-8e9a-7e03da870287"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:27.021572 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.021546 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7358276-038a-487d-8e9a-7e03da870287-kube-api-access-plxrt" pod "b7358276-038a-487d-8e9a-7e03da870287" (UID: "b7358276-038a-487d-8e9a-7e03da870287"). InnerVolumeSpecName "kube-api-access-plxrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:27.025120 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.025098 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" Oct 31 03:45:27.120413 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.120324 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240f9af4-2366-4fb8-b203-045e73da25ca-kube-api-access-hpc42\" (UniqueName: \"kubernetes.io/projected/240f9af4-2366-4fb8-b203-045e73da25ca-kube-api-access-hpc42\") pod \"240f9af4-2366-4fb8-b203-045e73da25ca\" (UID: \"240f9af4-2366-4fb8-b203-045e73da25ca\") " Oct 31 03:45:27.120628 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.120482 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/240f9af4-2366-4fb8-b203-045e73da25ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240f9af4-2366-4fb8-b203-045e73da25ca-maas-api-service-ca\") pod \"240f9af4-2366-4fb8-b203-045e73da25ca\" (UID: \"240f9af4-2366-4fb8-b203-045e73da25ca\") " Oct 31 03:45:27.120848 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.120821 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plxrt\" (UniqueName: \"kubernetes.io/projected/b7358276-038a-487d-8e9a-7e03da870287-kube-api-access-plxrt\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:27.120848 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.120850 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7358276-038a-487d-8e9a-7e03da870287-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:27.121115 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.121090 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240f9af4-2366-4fb8-b203-045e73da25ca-maas-api-service-ca" pod "240f9af4-2366-4fb8-b203-045e73da25ca" (UID: "240f9af4-2366-4fb8-b203-045e73da25ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:27.123783 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.123742 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240f9af4-2366-4fb8-b203-045e73da25ca-kube-api-access-hpc42" pod "240f9af4-2366-4fb8-b203-045e73da25ca" (UID: "240f9af4-2366-4fb8-b203-045e73da25ca"). InnerVolumeSpecName "kube-api-access-hpc42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:27.132517 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.132413 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" path="/var/lib/kubelet/pods/18e4b978-f8d2-4d70-8ac5-3777558e6ac3/volumes" Oct 31 03:45:27.132948 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.132911 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" path="/var/lib/kubelet/pods/cb3536eb-9c15-43fc-ba82-b7f143484438/volumes" Oct 31 03:45:27.222475 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.222445 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/240f9af4-2366-4fb8-b203-045e73da25ca-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:27.222475 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.222472 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpc42\" (UniqueName: \"kubernetes.io/projected/240f9af4-2366-4fb8-b203-045e73da25ca-kube-api-access-hpc42\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:27.436831 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.436804 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" Oct 31 03:45:27.436831 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.436824 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6" event={"ID":"b7358276-038a-487d-8e9a-7e03da870287","Type":"ContainerDied","Data":"584f5fbc1d4f718fb9e6cba8ee047feaf341c93408a81200751d09311e041bc7"} Oct 31 03:45:27.437393 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.436915 4426 scope.go:122] "RemoveContainer" containerID="c43bf6324cdfe8bada49a31ce632824dd998b88ec2701c57d3db9bdeeab7919c" Oct 31 03:45:27.441401 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.441368 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" event={"ID":"240f9af4-2366-4fb8-b203-045e73da25ca","Type":"ContainerDied","Data":"b3ebbcd74227c554b49d818939fb840ba154b5d8535095f1e054dad572ac2176"} Oct 31 03:45:27.441524 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.441499 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj" Oct 31 03:45:27.447633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.447612 4426 scope.go:122] "RemoveContainer" containerID="94c2a82cbbaba2e692e55188b8dee8d3d2abbc9bcdc602d00a46407a8321395e" Oct 31 03:45:27.453345 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.453317 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:45:27.596802 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.596767 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj"] Oct 31 03:45:27.600435 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.600401 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d7b7bc-29890305-c8nnj"] Oct 31 03:45:27.691218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.691107 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh"] Oct 31 03:45:27.703064 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.703029 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6"] Oct 31 03:45:27.707985 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.707954 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890305-29nx6"] Oct 31 03:45:27.834901 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.834874 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" Oct 31 03:45:27.930613 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.930577 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-maas-api-service-ca\") pod \"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3\" (UID: \"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3\") " Oct 31 03:45:27.930812 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.930629 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-kube-api-access-rcdkw\" (UniqueName: \"kubernetes.io/projected/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-kube-api-access-rcdkw\") pod \"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3\" (UID: \"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3\") " Oct 31 03:45:27.930956 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.930932 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-maas-api-service-ca" pod "8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" (UID: "8bf7bd53-33db-44bd-89cc-d60bbe65c1c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:27.932975 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:27.932944 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-kube-api-access-rcdkw" pod "8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" (UID: "8bf7bd53-33db-44bd-89cc-d60bbe65c1c3"). InnerVolumeSpecName "kube-api-access-rcdkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:28.032273 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:28.032185 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:28.032273 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:28.032227 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcdkw\" (UniqueName: \"kubernetes.io/projected/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3-kube-api-access-rcdkw\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:28.451593 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:28.451549 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" Oct 31 03:45:28.451593 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:28.451559 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh" event={"ID":"8bf7bd53-33db-44bd-89cc-d60bbe65c1c3","Type":"ContainerDied","Data":"15fd5fe630cf26f425b4e46a022db710800dab0d71290923531933388b95fb7b"} Oct 31 03:45:28.452211 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:28.451631 4426 scope.go:122] "RemoveContainer" containerID="1dc1062c9448cf7fc374f1989d9626a0dab3169f0c7d2f441e047c816a85686b" Oct 31 03:45:28.607835 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:28.607795 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh"] Oct 31 03:45:28.612102 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:28.612066 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890305-l76nh"] Oct 31 03:45:29.131504 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:29.131468 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" path="/var/lib/kubelet/pods/240f9af4-2366-4fb8-b203-045e73da25ca/volumes" Oct 31 03:45:29.132079 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:29.132062 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" path="/var/lib/kubelet/pods/8bf7bd53-33db-44bd-89cc-d60bbe65c1c3/volumes" Oct 31 03:45:29.132615 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:29.132597 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7358276-038a-487d-8e9a-7e03da870287" path="/var/lib/kubelet/pods/b7358276-038a-487d-8e9a-7e03da870287/volumes" Oct 31 03:45:32.848285 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:45:32.848249 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:45:33.465180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465150 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5"] Oct 31 03:45:33.465661 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465648 4426 cpu_manager.go:414] "containerMap: removing container" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerName="cleanup" Oct 31 03:45:33.465715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465662 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerName="cleanup" Oct 31 03:45:33.465715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465673 4426 cpu_manager.go:414] "containerMap: removing container" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" containerName="cleanup" Oct 31 03:45:33.465715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465677 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" containerName="cleanup" Oct 31 03:45:33.465715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465685 4426 cpu_manager.go:414] "containerMap: removing container" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerName="cleanup" Oct 31 03:45:33.465715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465690 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerName="cleanup" Oct 31 03:45:33.465715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465699 4426 cpu_manager.go:414] "containerMap: removing container" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" containerName="cleanup" Oct 31 03:45:33.465715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465703 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" containerName="cleanup" Oct 31 03:45:33.465715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465714 4426 cpu_manager.go:414] "containerMap: removing container" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerName="cleanup" Oct 31 03:45:33.465715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465718 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465724 4426 cpu_manager.go:414] "containerMap: removing container" podUID="b7358276-038a-487d-8e9a-7e03da870287" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465729 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7358276-038a-487d-8e9a-7e03da870287" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465744 4426 cpu_manager.go:414] "containerMap: removing container" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465748 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465764 4426 cpu_manager.go:414] "containerMap: removing container" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465769 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465775 4426 cpu_manager.go:414] "containerMap: removing container" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465779 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465784 4426 cpu_manager.go:414] "containerMap: removing container" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465788 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465794 4426 cpu_manager.go:414] "containerMap: removing container" podUID="b7358276-038a-487d-8e9a-7e03da870287" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465798 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7358276-038a-487d-8e9a-7e03da870287" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465868 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465875 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465881 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465886 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465892 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465897 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465901 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7358276-038a-487d-8e9a-7e03da870287" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465909 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465915 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb991d3b-1510-4813-bc64-8c5c61c9614e" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465921 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7358276-038a-487d-8e9a-7e03da870287" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465927 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465932 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465938 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerName="cleanup" Oct 31 03:45:33.465986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.465944 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerName="cleanup" Oct 31 03:45:33.470916 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.470892 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.526301 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.526265 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5"] Oct 31 03:45:33.567508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.567467 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a64fd2\"/\"kube-root-ca.crt\"" Oct 31 03:45:33.567508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.567479 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a64fd2\"/\"simulator-w2-a64fd2-kserve-self-signed-certs\"" Oct 31 03:45:33.567773 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.567594 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a64fd2\"/\"openshift-service-ca.crt\"" Oct 31 03:45:33.567773 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.567686 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a64fd2\"/\"default-dockercfg-xrsfg\"" Oct 31 03:45:33.599466 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.599428 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tmp-dir\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.599641 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.599553 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tls-certs\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.599641 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.599596 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-model-cache\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.599735 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.599656 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-home\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.599735 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.599716 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj9zc\" (UniqueName: \"kubernetes.io/projected/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-kube-api-access-vj9zc\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.599824 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.599806 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-dshm\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.701250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.701220 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tls-certs\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.701400 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.701262 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-model-cache\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.701400 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.701321 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-home\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.701400 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.701341 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj9zc\" (UniqueName: \"kubernetes.io/projected/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-kube-api-access-vj9zc\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.701400 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.701388 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-dshm\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.701629 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.701439 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tmp-dir\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.701806 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.701781 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-home\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.701889 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.701874 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-model-cache\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.701948 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.701909 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tmp-dir\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.703645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.703618 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-dshm\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.703888 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.703870 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tls-certs\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.761664 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.761566 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj9zc\" (UniqueName: \"kubernetes.io/projected/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-kube-api-access-vj9zc\") pod \"simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:33.783563 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:33.783520 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:34.162685 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:34.162648 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5"] Oct 31 03:45:34.163937 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:45:34.163914 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod638fac5f_cc53_4bb2_b6c8_6fd2f0ace35c.slice/crio-80d34aab70e9b446c4f5c03820fdc6088b0f59609620c52e37c4ea02b36b1d21 WatchSource:0}: Error finding container 80d34aab70e9b446c4f5c03820fdc6088b0f59609620c52e37c4ea02b36b1d21: Status 404 returned error can't find the container with id 80d34aab70e9b446c4f5c03820fdc6088b0f59609620c52e37c4ea02b36b1d21 Oct 31 03:45:34.366002 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:34.365971 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:34.378891 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:34.378861 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:34.507600 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:34.507509 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" event={"ID":"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c","Type":"ContainerStarted","Data":"bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360"} Oct 31 03:45:34.507600 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:34.507559 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" event={"ID":"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c","Type":"ContainerStarted","Data":"80d34aab70e9b446c4f5c03820fdc6088b0f59609620c52e37c4ea02b36b1d21"} Oct 31 03:45:34.519180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:34.519155 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:45:34.617521 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:34.617463 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podStartSLOduration=1.617448318 podStartE2EDuration="1.617448318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:34.617194035 +0000 UTC m=+1031.617819262" watchObservedRunningTime="2026-10-31 03:45:34.617448318 +0000 UTC m=+1031.618073546" Oct 31 03:45:34.784504 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:34.784427 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:39.660844 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:39.660812 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:45:43.783843 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:43.783787 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:43.801453 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:43.801422 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:44.594142 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:44.594107 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:45:50.780598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.780552 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24"] Oct 31 03:45:50.781180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781159 4426 cpu_manager.go:414] "containerMap: removing container" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerName="cleanup" Oct 31 03:45:50.781180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781178 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3536eb-9c15-43fc-ba82-b7f143484438" containerName="cleanup" Oct 31 03:45:50.781180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781189 4426 cpu_manager.go:414] "containerMap: removing container" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerName="cleanup" Oct 31 03:45:50.781180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781195 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e4b978-f8d2-4d70-8ac5-3777558e6ac3" containerName="cleanup" Oct 31 03:45:50.781408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781219 4426 cpu_manager.go:414] "containerMap: removing container" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" containerName="cleanup" Oct 31 03:45:50.781408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781225 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" containerName="cleanup" Oct 31 03:45:50.781408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781236 4426 cpu_manager.go:414] "containerMap: removing container" podUID="b7358276-038a-487d-8e9a-7e03da870287" containerName="cleanup" Oct 31 03:45:50.781408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781242 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7358276-038a-487d-8e9a-7e03da870287" containerName="cleanup" Oct 31 03:45:50.781408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781259 4426 cpu_manager.go:414] "containerMap: removing container" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerName="cleanup" Oct 31 03:45:50.781408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781264 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf7bd53-33db-44bd-89cc-d60bbe65c1c3" containerName="cleanup" Oct 31 03:45:50.781408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781353 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="240f9af4-2366-4fb8-b203-045e73da25ca" containerName="cleanup" Oct 31 03:45:50.781408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.781365 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7358276-038a-487d-8e9a-7e03da870287" containerName="cleanup" Oct 31 03:45:50.786530 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.786512 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.802019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.801990 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrtm\" (UniqueName: \"kubernetes.io/projected/1d00ff31-e8c5-4898-84d8-a4ff023846fe-kube-api-access-8vrtm\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.802188 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.802028 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tls-certs\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.802188 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.802106 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-model-cache\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.802188 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.802175 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-dshm\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.802305 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.802210 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tmp-dir\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.802305 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.802241 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-home\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.856264 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.856216 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24"] Oct 31 03:45:50.883062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.883016 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-537e34\"/\"kube-root-ca.crt\"" Oct 31 03:45:50.883300 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.883140 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-537e34\"/\"default-dockercfg-9fnbs\"" Oct 31 03:45:50.883300 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.883171 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-537e34\"/\"openshift-service-ca.crt\"" Oct 31 03:45:50.883300 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.883254 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-537e34\"/\"premium-w3-537e34-kserve-self-signed-certs\"" Oct 31 03:45:50.903393 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.903356 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tls-certs\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.903565 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.903404 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-model-cache\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.903565 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.903524 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-dshm\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.903641 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.903574 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tmp-dir\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.903641 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.903612 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-home\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.903728 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.903710 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrtm\" (UniqueName: \"kubernetes.io/projected/1d00ff31-e8c5-4898-84d8-a4ff023846fe-kube-api-access-8vrtm\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.903831 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.903811 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-model-cache\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.904034 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.904003 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tmp-dir\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.904153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.904040 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-home\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.905723 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.905703 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-dshm\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.905995 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.905979 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tls-certs\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:50.961980 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:50.961938 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrtm\" (UniqueName: \"kubernetes.io/projected/1d00ff31-e8c5-4898-84d8-a4ff023846fe-kube-api-access-8vrtm\") pod \"premium-w3-537e34-kserve-68668dd94-hdf24\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:51.099596 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:51.099497 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:45:51.505796 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:51.505764 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24"] Oct 31 03:45:51.507441 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:45:51.507412 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d00ff31_e8c5_4898_84d8_a4ff023846fe.slice/crio-910134076d328ae0647fe04c38c35501df20a5fc1dfb215009356866daaca9e2 WatchSource:0}: Error finding container 910134076d328ae0647fe04c38c35501df20a5fc1dfb215009356866daaca9e2: Status 404 returned error can't find the container with id 910134076d328ae0647fe04c38c35501df20a5fc1dfb215009356866daaca9e2 Oct 31 03:45:51.634714 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:51.634671 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" event={"ID":"1d00ff31-e8c5-4898-84d8-a4ff023846fe","Type":"ContainerStarted","Data":"aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600"} Oct 31 03:45:51.634886 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:51.634726 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" event={"ID":"1d00ff31-e8c5-4898-84d8-a4ff023846fe","Type":"ContainerStarted","Data":"910134076d328ae0647fe04c38c35501df20a5fc1dfb215009356866daaca9e2"} Oct 31 03:45:51.749554 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:51.749486 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podStartSLOduration=1.749464431 podStartE2EDuration="1.749464431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:51.744793862 +0000 UTC m=+1048.745419091" watchObservedRunningTime="2026-10-31 03:45:51.749464431 +0000 UTC m=+1048.750089660" Oct 31 03:45:52.100288 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:45:52.100240 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:46:01.100053 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:01.100012 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:46:01.112842 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:01.112804 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:46:01.717035 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:01.716990 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:46:02.847635 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:46:02.847070 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:46:03.897954 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:03.897912 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:46:20.098093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.098051 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr"] Oct 31 03:46:20.101759 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.101740 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.135077 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.135046 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-home\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.135279 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.135097 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-tmp-dir\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.135279 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.135192 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-model-cache\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.135279 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.135232 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9810df30-5ed6-4aaf-9ec9-e25106217852-tls-certs\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.135451 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.135349 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-dshm\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.135451 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.135436 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmzn\" (UniqueName: \"kubernetes.io/projected/9810df30-5ed6-4aaf-9ec9-e25106217852-kube-api-access-7zmzn\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.164593 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.164553 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr"] Oct 31 03:46:20.200111 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.200056 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d7b7bc\"/\"openshift-service-ca.crt\"" Oct 31 03:46:20.200111 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.200056 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d7b7bc\"/\"default-dockercfg-dfk9z\"" Oct 31 03:46:20.200381 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.200088 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d7b7bc\"/\"kube-root-ca.crt\"" Oct 31 03:46:20.200381 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.200231 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d7b7bc\"/\"premium-w1-d7b7bc-kserve-self-signed-certs\"" Oct 31 03:46:20.256147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.236893 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-dshm\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.256147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.237011 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmzn\" (UniqueName: \"kubernetes.io/projected/9810df30-5ed6-4aaf-9ec9-e25106217852-kube-api-access-7zmzn\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.256147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.237074 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-home\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.256147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.237175 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-tmp-dir\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.256147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.237204 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-model-cache\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.256147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.237241 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9810df30-5ed6-4aaf-9ec9-e25106217852-tls-certs\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.256147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.237935 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-model-cache\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.256147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.240439 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-tmp-dir\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.256147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.240458 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9810df30-5ed6-4aaf-9ec9-e25106217852-tls-certs\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.256147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.242951 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-dshm\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.256147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.250850 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-home\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.295254 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.295219 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmzn\" (UniqueName: \"kubernetes.io/projected/9810df30-5ed6-4aaf-9ec9-e25106217852-kube-api-access-7zmzn\") pod \"premium-w1-d7b7bc-kserve-78c6668ffd-72dqr\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.415883 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.415842 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:20.807537 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.807505 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr"] Oct 31 03:46:20.808767 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:46:20.808738 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9810df30_5ed6_4aaf_9ec9_e25106217852.slice/crio-aca35e041516798d836b38c6842d8f82447b9b8421ce58fa2f8ca79106ff46ae WatchSource:0}: Error finding container aca35e041516798d836b38c6842d8f82447b9b8421ce58fa2f8ca79106ff46ae: Status 404 returned error can't find the container with id aca35e041516798d836b38c6842d8f82447b9b8421ce58fa2f8ca79106ff46ae Oct 31 03:46:20.845908 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:20.845870 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" event={"ID":"9810df30-5ed6-4aaf-9ec9-e25106217852","Type":"ContainerStarted","Data":"aca35e041516798d836b38c6842d8f82447b9b8421ce58fa2f8ca79106ff46ae"} Oct 31 03:46:21.856334 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:21.856284 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" event={"ID":"9810df30-5ed6-4aaf-9ec9-e25106217852","Type":"ContainerStarted","Data":"19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0"} Oct 31 03:46:21.967836 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:21.967778 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podStartSLOduration=1.9677621699999999 podStartE2EDuration="1.96776217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:21.965630472 +0000 UTC m=+1078.966255701" watchObservedRunningTime="2026-10-31 03:46:21.96776217 +0000 UTC m=+1078.968387394" Oct 31 03:46:22.416222 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:22.416177 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:27.979675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:27.979633 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:46:28.181554 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.181516 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2"] Oct 31 03:46:28.187679 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.187651 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:28.230187 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.230063 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cec0039-9ztt2\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:28.230187 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.230103 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mxsj\" (UniqueName: \"kubernetes.io/projected/c7a4c766-3923-4dd7-84bc-b338430976d4-kube-api-access-8mxsj\") pod \"maas-api-revoke-sub-6cec0039-9ztt2\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:28.230187 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.230137 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cec0039-9ztt2\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:28.239861 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.239826 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:46:28.241983 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.241940 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2"] Oct 31 03:46:28.283590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.283544 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:46:28.283590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.283598 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:46:28.283858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.283604 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:46:28.331466 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.331426 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cec0039-9ztt2\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:28.331466 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.331467 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mxsj\" (UniqueName: \"kubernetes.io/projected/c7a4c766-3923-4dd7-84bc-b338430976d4-kube-api-access-8mxsj\") pod \"maas-api-revoke-sub-6cec0039-9ztt2\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:28.331678 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.331499 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cec0039-9ztt2\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:28.332461 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.332441 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cec0039-9ztt2\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:28.333982 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.333944 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cec0039-9ztt2\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:28.412158 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.405016 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mxsj\" (UniqueName: \"kubernetes.io/projected/c7a4c766-3923-4dd7-84bc-b338430976d4-kube-api-access-8mxsj\") pod \"maas-api-revoke-sub-6cec0039-9ztt2\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:28.503328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.503241 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:28.882037 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.882007 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2"] Oct 31 03:46:28.883487 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:46:28.883450 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a4c766_3923_4dd7_84bc_b338430976d4.slice/crio-b3e0c2365f10ee788f767d8538343a537446e0c2b0f248094c8a1be2c499f7f7 WatchSource:0}: Error finding container b3e0c2365f10ee788f767d8538343a537446e0c2b0f248094c8a1be2c499f7f7: Status 404 returned error can't find the container with id b3e0c2365f10ee788f767d8538343a537446e0c2b0f248094c8a1be2c499f7f7 Oct 31 03:46:28.911326 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:28.911294 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" event={"ID":"c7a4c766-3923-4dd7-84bc-b338430976d4","Type":"ContainerStarted","Data":"b3e0c2365f10ee788f767d8538343a537446e0c2b0f248094c8a1be2c499f7f7"} Oct 31 03:46:29.918835 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:29.918794 4426 generic.go:356] "Generic (PLEG): container finished" podID="c7a4c766-3923-4dd7-84bc-b338430976d4" containerID="342d02f6c0432dbcab5bfa9c68b8702eb3721396517dd467f06e795611524fd6" exitCode=0 Oct 31 03:46:29.919262 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:29.918845 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" event={"ID":"c7a4c766-3923-4dd7-84bc-b338430976d4","Type":"ContainerDied","Data":"342d02f6c0432dbcab5bfa9c68b8702eb3721396517dd467f06e795611524fd6"} Oct 31 03:46:30.416302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:30.416262 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:30.429180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:30.429117 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:30.944998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:30.944718 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:46:31.088419 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.088375 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:31.161530 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.161478 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-xks-ca\") pod \"c7a4c766-3923-4dd7-84bc-b338430976d4\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " Oct 31 03:46:31.161708 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.161552 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a4c766-3923-4dd7-84bc-b338430976d4-kube-api-access-8mxsj\" (UniqueName: \"kubernetes.io/projected/c7a4c766-3923-4dd7-84bc-b338430976d4-kube-api-access-8mxsj\") pod \"c7a4c766-3923-4dd7-84bc-b338430976d4\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " Oct 31 03:46:31.161792 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.161717 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-openshift-ca\") pod \"c7a4c766-3923-4dd7-84bc-b338430976d4\" (UID: \"c7a4c766-3923-4dd7-84bc-b338430976d4\") " Oct 31 03:46:31.162175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.162081 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-openshift-ca" pod "c7a4c766-3923-4dd7-84bc-b338430976d4" (UID: "c7a4c766-3923-4dd7-84bc-b338430976d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:31.162317 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.162222 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:31.163741 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.163711 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-xks-ca" pod "c7a4c766-3923-4dd7-84bc-b338430976d4" (UID: "c7a4c766-3923-4dd7-84bc-b338430976d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:46:31.163885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.163853 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a4c766-3923-4dd7-84bc-b338430976d4-kube-api-access-8mxsj" pod "c7a4c766-3923-4dd7-84bc-b338430976d4" (UID: "c7a4c766-3923-4dd7-84bc-b338430976d4"). InnerVolumeSpecName "kube-api-access-8mxsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:31.263286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.263199 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a4c766-3923-4dd7-84bc-b338430976d4-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:31.263286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.263229 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mxsj\" (UniqueName: \"kubernetes.io/projected/c7a4c766-3923-4dd7-84bc-b338430976d4-kube-api-access-8mxsj\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:31.937197 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.937146 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" Oct 31 03:46:31.937197 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.937158 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2" event={"ID":"c7a4c766-3923-4dd7-84bc-b338430976d4","Type":"ContainerDied","Data":"b3e0c2365f10ee788f767d8538343a537446e0c2b0f248094c8a1be2c499f7f7"} Oct 31 03:46:31.937491 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:31.937218 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3e0c2365f10ee788f767d8538343a537446e0c2b0f248094c8a1be2c499f7f7" Oct 31 03:46:32.851672 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:46:32.851632 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:46:43.752271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.752233 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn"] Oct 31 03:46:43.752789 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.752774 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c7a4c766-3923-4dd7-84bc-b338430976d4" containerName="revoke-keys" Oct 31 03:46:43.752833 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.752790 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a4c766-3923-4dd7-84bc-b338430976d4" containerName="revoke-keys" Oct 31 03:46:43.752916 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.752906 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a4c766-3923-4dd7-84bc-b338430976d4" containerName="revoke-keys" Oct 31 03:46:43.756406 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.756375 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:43.805490 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.805457 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:46:43.851837 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.851809 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:46:43.852075 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.851830 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:46:43.852228 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.851967 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:46:43.857319 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.857282 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn"] Oct 31 03:46:43.908368 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.908327 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353a4c3c-4bksn\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:43.908529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.908381 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6fkz\" (UniqueName: \"kubernetes.io/projected/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-kube-api-access-s6fkz\") pod \"maas-api-revoke-sub-353a4c3c-4bksn\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:43.908529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.908483 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353a4c3c-4bksn\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:43.986466 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.986429 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6"] Oct 31 03:46:43.990490 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:43.990470 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:44.010076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.009998 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6fkz\" (UniqueName: \"kubernetes.io/projected/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-kube-api-access-s6fkz\") pod \"maas-api-revoke-sub-353a4c3c-4bksn\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:44.010076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.010049 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353a4c3c-4bksn\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:44.010281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.010189 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353a4c3c-4bksn\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:44.010797 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.010779 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353a4c3c-4bksn\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:44.012442 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.012419 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353a4c3c-4bksn\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:44.048840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.048800 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6"] Oct 31 03:46:44.068409 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.068378 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6fkz\" (UniqueName: \"kubernetes.io/projected/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-kube-api-access-s6fkz\") pod \"maas-api-revoke-sub-353a4c3c-4bksn\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:44.070206 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.070178 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:44.111399 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.111351 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg7wm\" (UniqueName: \"kubernetes.io/projected/02c94bd5-1985-42be-ae63-0004dfcddfde-kube-api-access-gg7wm\") pod \"maas-api-revoke-sub-8ca26021-9djp6\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:44.111571 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.111413 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ca26021-9djp6\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:44.111627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.111569 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ca26021-9djp6\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:44.212705 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.212670 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ca26021-9djp6\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:44.212867 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.212820 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ca26021-9djp6\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:44.212940 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.212928 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg7wm\" (UniqueName: \"kubernetes.io/projected/02c94bd5-1985-42be-ae63-0004dfcddfde-kube-api-access-gg7wm\") pod \"maas-api-revoke-sub-8ca26021-9djp6\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:44.213515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.213488 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ca26021-9djp6\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:44.215292 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.215262 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ca26021-9djp6\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:44.270933 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.270859 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg7wm\" (UniqueName: \"kubernetes.io/projected/02c94bd5-1985-42be-ae63-0004dfcddfde-kube-api-access-gg7wm\") pod \"maas-api-revoke-sub-8ca26021-9djp6\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:44.303750 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.303717 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:44.455358 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.455319 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn"] Oct 31 03:46:44.457592 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:46:44.457561 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc49eec_476b_4c81_8d47_9f14cc4ce77c.slice/crio-e66f8864aff7b39d07b85410fa85530a0473a26012e3c3646b5fdcde5b4a11b1 WatchSource:0}: Error finding container e66f8864aff7b39d07b85410fa85530a0473a26012e3c3646b5fdcde5b4a11b1: Status 404 returned error can't find the container with id e66f8864aff7b39d07b85410fa85530a0473a26012e3c3646b5fdcde5b4a11b1 Oct 31 03:46:44.701830 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:44.701789 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6"] Oct 31 03:46:44.703769 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:46:44.703743 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c94bd5_1985_42be_ae63_0004dfcddfde.slice/crio-fad06607ff56dc1663687ed02b6efd16d92cbbd00a70320af5f9d21af603fb1e WatchSource:0}: Error finding container fad06607ff56dc1663687ed02b6efd16d92cbbd00a70320af5f9d21af603fb1e: Status 404 returned error can't find the container with id fad06607ff56dc1663687ed02b6efd16d92cbbd00a70320af5f9d21af603fb1e Oct 31 03:46:45.038286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:45.038255 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" event={"ID":"02c94bd5-1985-42be-ae63-0004dfcddfde","Type":"ContainerStarted","Data":"7916937100f423512b0dc8e4afd1f09bf5aee29cffbb63c0751315db775b9c14"} Oct 31 03:46:45.038726 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:45.038322 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" event={"ID":"02c94bd5-1985-42be-ae63-0004dfcddfde","Type":"ContainerStarted","Data":"fad06607ff56dc1663687ed02b6efd16d92cbbd00a70320af5f9d21af603fb1e"} Oct 31 03:46:45.040108 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:45.040081 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" event={"ID":"bbc49eec-476b-4c81-8d47-9f14cc4ce77c","Type":"ContainerStarted","Data":"fdac12a574aba06f3aa2daf0c9f07ffe7a3f52840d579aa04ed59fd03517bec3"} Oct 31 03:46:45.040238 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:45.040141 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" event={"ID":"bbc49eec-476b-4c81-8d47-9f14cc4ce77c","Type":"ContainerStarted","Data":"e66f8864aff7b39d07b85410fa85530a0473a26012e3c3646b5fdcde5b4a11b1"} Oct 31 03:46:45.149142 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:45.149077 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" podStartSLOduration=2.149063097 podStartE2EDuration="2.149063097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:45.146235078 +0000 UTC m=+1102.146860306" watchObservedRunningTime="2026-10-31 03:46:45.149063097 +0000 UTC m=+1102.149688325" Oct 31 03:46:45.257108 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:45.256998 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" podStartSLOduration=2.256982638 podStartE2EDuration="2.256982638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:45.254851485 +0000 UTC m=+1102.255476714" watchObservedRunningTime="2026-10-31 03:46:45.256982638 +0000 UTC m=+1102.257607879" Oct 31 03:46:46.049003 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:46.048913 4426 generic.go:356] "Generic (PLEG): container finished" podID="02c94bd5-1985-42be-ae63-0004dfcddfde" containerID="7916937100f423512b0dc8e4afd1f09bf5aee29cffbb63c0751315db775b9c14" exitCode=0 Oct 31 03:46:46.049437 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:46.049029 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" event={"ID":"02c94bd5-1985-42be-ae63-0004dfcddfde","Type":"ContainerDied","Data":"7916937100f423512b0dc8e4afd1f09bf5aee29cffbb63c0751315db775b9c14"} Oct 31 03:46:46.050909 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:46.050879 4426 generic.go:356] "Generic (PLEG): container finished" podID="bbc49eec-476b-4c81-8d47-9f14cc4ce77c" containerID="fdac12a574aba06f3aa2daf0c9f07ffe7a3f52840d579aa04ed59fd03517bec3" exitCode=0 Oct 31 03:46:46.051039 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:46.050923 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" event={"ID":"bbc49eec-476b-4c81-8d47-9f14cc4ce77c","Type":"ContainerDied","Data":"fdac12a574aba06f3aa2daf0c9f07ffe7a3f52840d579aa04ed59fd03517bec3"} Oct 31 03:46:47.341920 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.341589 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:47.386773 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.386744 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:47.476146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.471884 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-openshift-ca\") pod \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " Oct 31 03:46:47.476146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.472002 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-xks-ca\") pod \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " Oct 31 03:46:47.476146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.472147 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-kube-api-access-s6fkz\" (UniqueName: \"kubernetes.io/projected/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-kube-api-access-s6fkz\") pod \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\" (UID: \"bbc49eec-476b-4c81-8d47-9f14cc4ce77c\") " Oct 31 03:46:47.476146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.472623 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-openshift-ca" pod "bbc49eec-476b-4c81-8d47-9f14cc4ce77c" (UID: "bbc49eec-476b-4c81-8d47-9f14cc4ce77c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:47.481744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.481701 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-kube-api-access-s6fkz" pod "bbc49eec-476b-4c81-8d47-9f14cc4ce77c" (UID: "bbc49eec-476b-4c81-8d47-9f14cc4ce77c"). InnerVolumeSpecName "kube-api-access-s6fkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:47.483658 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.483614 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-xks-ca" pod "bbc49eec-476b-4c81-8d47-9f14cc4ce77c" (UID: "bbc49eec-476b-4c81-8d47-9f14cc4ce77c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:46:47.573509 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.573410 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-openshift-ca\") pod \"02c94bd5-1985-42be-ae63-0004dfcddfde\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " Oct 31 03:46:47.573509 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.573452 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-xks-ca\") pod \"02c94bd5-1985-42be-ae63-0004dfcddfde\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " Oct 31 03:46:47.573741 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.573554 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c94bd5-1985-42be-ae63-0004dfcddfde-kube-api-access-gg7wm\" (UniqueName: \"kubernetes.io/projected/02c94bd5-1985-42be-ae63-0004dfcddfde-kube-api-access-gg7wm\") pod \"02c94bd5-1985-42be-ae63-0004dfcddfde\" (UID: \"02c94bd5-1985-42be-ae63-0004dfcddfde\") " Oct 31 03:46:47.573842 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.573816 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:47.573842 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.573833 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:47.573842 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.573842 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6fkz\" (UniqueName: \"kubernetes.io/projected/bbc49eec-476b-4c81-8d47-9f14cc4ce77c-kube-api-access-s6fkz\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:47.573995 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.573854 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-openshift-ca" pod "02c94bd5-1985-42be-ae63-0004dfcddfde" (UID: "02c94bd5-1985-42be-ae63-0004dfcddfde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:47.575640 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.575610 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-xks-ca" pod "02c94bd5-1985-42be-ae63-0004dfcddfde" (UID: "02c94bd5-1985-42be-ae63-0004dfcddfde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:46:47.575796 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.575683 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c94bd5-1985-42be-ae63-0004dfcddfde-kube-api-access-gg7wm" pod "02c94bd5-1985-42be-ae63-0004dfcddfde" (UID: "02c94bd5-1985-42be-ae63-0004dfcddfde"). InnerVolumeSpecName "kube-api-access-gg7wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:47.675259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.675223 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg7wm\" (UniqueName: \"kubernetes.io/projected/02c94bd5-1985-42be-ae63-0004dfcddfde-kube-api-access-gg7wm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:47.675259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.675255 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:47.675259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:47.675265 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c94bd5-1985-42be-ae63-0004dfcddfde-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:48.069701 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:48.069663 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" Oct 31 03:46:48.069701 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:48.069682 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6" event={"ID":"02c94bd5-1985-42be-ae63-0004dfcddfde","Type":"ContainerDied","Data":"fad06607ff56dc1663687ed02b6efd16d92cbbd00a70320af5f9d21af603fb1e"} Oct 31 03:46:48.069983 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:48.069731 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad06607ff56dc1663687ed02b6efd16d92cbbd00a70320af5f9d21af603fb1e" Oct 31 03:46:48.071299 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:48.071275 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" Oct 31 03:46:48.071421 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:48.071307 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn" event={"ID":"bbc49eec-476b-4c81-8d47-9f14cc4ce77c","Type":"ContainerDied","Data":"e66f8864aff7b39d07b85410fa85530a0473a26012e3c3646b5fdcde5b4a11b1"} Oct 31 03:46:48.071421 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:48.071355 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e66f8864aff7b39d07b85410fa85530a0473a26012e3c3646b5fdcde5b4a11b1" Oct 31 03:46:59.559524 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.559485 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h"] Oct 31 03:46:59.560025 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.560008 4426 cpu_manager.go:414] "containerMap: removing container" podUID="02c94bd5-1985-42be-ae63-0004dfcddfde" containerName="revoke-keys" Oct 31 03:46:59.560025 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.560020 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c94bd5-1985-42be-ae63-0004dfcddfde" containerName="revoke-keys" Oct 31 03:46:59.560096 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.560048 4426 cpu_manager.go:414] "containerMap: removing container" podUID="bbc49eec-476b-4c81-8d47-9f14cc4ce77c" containerName="revoke-keys" Oct 31 03:46:59.560096 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.560058 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc49eec-476b-4c81-8d47-9f14cc4ce77c" containerName="revoke-keys" Oct 31 03:46:59.560181 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.560161 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc49eec-476b-4c81-8d47-9f14cc4ce77c" containerName="revoke-keys" Oct 31 03:46:59.560181 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.560173 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c94bd5-1985-42be-ae63-0004dfcddfde" containerName="revoke-keys" Oct 31 03:46:59.565175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.565157 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:46:59.603072 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.603040 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktv7t\" (UniqueName: \"kubernetes.io/projected/773296c4-147e-48e7-876b-f07250adb3f0-kube-api-access-ktv7t\") pod \"maas-api-revoke-sub-5709b7d5-5qf2h\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:46:59.603239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.603119 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5709b7d5-5qf2h\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:46:59.603239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.603202 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5709b7d5-5qf2h\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:46:59.620668 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.620635 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h"] Oct 31 03:46:59.662334 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.662305 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:46:59.662493 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.662357 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:46:59.662493 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.662357 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:46:59.662493 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.662379 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:46:59.704020 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.703989 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktv7t\" (UniqueName: \"kubernetes.io/projected/773296c4-147e-48e7-876b-f07250adb3f0-kube-api-access-ktv7t\") pod \"maas-api-revoke-sub-5709b7d5-5qf2h\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:46:59.704249 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.704052 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5709b7d5-5qf2h\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:46:59.704249 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.704084 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5709b7d5-5qf2h\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:46:59.704767 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.704744 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5709b7d5-5qf2h\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:46:59.706605 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.706576 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5709b7d5-5qf2h\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:46:59.761876 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.761845 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktv7t\" (UniqueName: \"kubernetes.io/projected/773296c4-147e-48e7-876b-f07250adb3f0-kube-api-access-ktv7t\") pod \"maas-api-revoke-sub-5709b7d5-5qf2h\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:46:59.878830 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:46:59.878743 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:47:00.248960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:00.248934 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h"] Oct 31 03:47:00.250199 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:47:00.250170 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773296c4_147e_48e7_876b_f07250adb3f0.slice/crio-c4123d7ff4efe739f1533a7c5f77ce1b25800bcabc12f5b1c7755f7b5afe231c WatchSource:0}: Error finding container c4123d7ff4efe739f1533a7c5f77ce1b25800bcabc12f5b1c7755f7b5afe231c: Status 404 returned error can't find the container with id c4123d7ff4efe739f1533a7c5f77ce1b25800bcabc12f5b1c7755f7b5afe231c Oct 31 03:47:00.797529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:00.797439 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2"] Oct 31 03:47:00.801611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:00.801586 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:00.851285 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:00.851255 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a64fd2\"/\"premium-w2-a64fd2-kserve-self-signed-certs\"" Oct 31 03:47:00.852771 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:00.852744 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2"] Oct 31 03:47:00.917981 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:00.917940 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-tmp-dir\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:00.917981 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:00.917986 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-dshm\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:00.918231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:00.918059 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlwsr\" (UniqueName: \"kubernetes.io/projected/3f5f78d0-b661-4846-a078-e80aefc88592-kube-api-access-hlwsr\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:00.918231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:00.918104 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-model-cache\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:00.918231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:00.918204 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5f78d0-b661-4846-a078-e80aefc88592-tls-certs\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:00.918373 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:00.918257 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-home\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.019718 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.019686 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-tmp-dir\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.019904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.019727 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-dshm\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.019904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.019761 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlwsr\" (UniqueName: \"kubernetes.io/projected/3f5f78d0-b661-4846-a078-e80aefc88592-kube-api-access-hlwsr\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.019904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.019805 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-model-cache\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.019904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.019843 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5f78d0-b661-4846-a078-e80aefc88592-tls-certs\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.019904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.019875 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-home\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.020253 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.020226 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-tmp-dir\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.020317 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.020287 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-home\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.020457 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.020435 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-model-cache\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.022115 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.022091 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-dshm\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.022887 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.022866 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5f78d0-b661-4846-a078-e80aefc88592-tls-certs\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.078030 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.077944 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlwsr\" (UniqueName: \"kubernetes.io/projected/3f5f78d0-b661-4846-a078-e80aefc88592-kube-api-access-hlwsr\") pod \"premium-w2-a64fd2-kserve-7954d4f997-gwqc2\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.117350 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.117306 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:01.190305 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.190270 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" event={"ID":"773296c4-147e-48e7-876b-f07250adb3f0","Type":"ContainerStarted","Data":"64db89033956adbf471421cedcb40b7c3a8fa8d747510f3fda4252b7626fd40d"} Oct 31 03:47:01.190481 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.190346 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" event={"ID":"773296c4-147e-48e7-876b-f07250adb3f0","Type":"ContainerStarted","Data":"c4123d7ff4efe739f1533a7c5f77ce1b25800bcabc12f5b1c7755f7b5afe231c"} Oct 31 03:47:01.301646 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.301580 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" podStartSLOduration=2.301566521 podStartE2EDuration="2.301566521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:01.298869295 +0000 UTC m=+1118.299494535" watchObservedRunningTime="2026-10-31 03:47:01.301566521 +0000 UTC m=+1118.302191785" Oct 31 03:47:01.515303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:01.515268 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2"] Oct 31 03:47:01.515987 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:47:01.515957 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f5f78d0_b661_4846_a078_e80aefc88592.slice/crio-1668cb5e77ed1a706415b12716e695ff6d6d77b9670beb037ff61c111c92654a WatchSource:0}: Error finding container 1668cb5e77ed1a706415b12716e695ff6d6d77b9670beb037ff61c111c92654a: Status 404 returned error can't find the container with id 1668cb5e77ed1a706415b12716e695ff6d6d77b9670beb037ff61c111c92654a Oct 31 03:47:02.199272 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:02.199234 4426 generic.go:356] "Generic (PLEG): container finished" podID="773296c4-147e-48e7-876b-f07250adb3f0" containerID="64db89033956adbf471421cedcb40b7c3a8fa8d747510f3fda4252b7626fd40d" exitCode=0 Oct 31 03:47:02.199695 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:02.199313 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" event={"ID":"773296c4-147e-48e7-876b-f07250adb3f0","Type":"ContainerDied","Data":"64db89033956adbf471421cedcb40b7c3a8fa8d747510f3fda4252b7626fd40d"} Oct 31 03:47:02.200971 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:02.200945 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" event={"ID":"3f5f78d0-b661-4846-a078-e80aefc88592","Type":"ContainerStarted","Data":"7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e"} Oct 31 03:47:02.201094 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:02.200975 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" event={"ID":"3f5f78d0-b661-4846-a078-e80aefc88592","Type":"ContainerStarted","Data":"1668cb5e77ed1a706415b12716e695ff6d6d77b9670beb037ff61c111c92654a"} Oct 31 03:47:02.420403 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:02.420347 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podStartSLOduration=2.420332312 podStartE2EDuration="2.420332312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:02.417555143 +0000 UTC m=+1119.418180371" watchObservedRunningTime="2026-10-31 03:47:02.420332312 +0000 UTC m=+1119.420957539" Oct 31 03:47:02.845967 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:47:02.845928 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:47:03.118528 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:03.118440 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:03.365636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:03.365608 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:47:03.446670 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:03.446637 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-openshift-ca\") pod \"773296c4-147e-48e7-876b-f07250adb3f0\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " Oct 31 03:47:03.446858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:03.446749 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-xks-ca\") pod \"773296c4-147e-48e7-876b-f07250adb3f0\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " Oct 31 03:47:03.446858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:03.446821 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773296c4-147e-48e7-876b-f07250adb3f0-kube-api-access-ktv7t\" (UniqueName: \"kubernetes.io/projected/773296c4-147e-48e7-876b-f07250adb3f0-kube-api-access-ktv7t\") pod \"773296c4-147e-48e7-876b-f07250adb3f0\" (UID: \"773296c4-147e-48e7-876b-f07250adb3f0\") " Oct 31 03:47:03.447025 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:03.447001 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-openshift-ca" pod "773296c4-147e-48e7-876b-f07250adb3f0" (UID: "773296c4-147e-48e7-876b-f07250adb3f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:03.447158 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:03.447144 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:03.448956 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:03.448928 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-xks-ca" pod "773296c4-147e-48e7-876b-f07250adb3f0" (UID: "773296c4-147e-48e7-876b-f07250adb3f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:03.449026 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:03.448946 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773296c4-147e-48e7-876b-f07250adb3f0-kube-api-access-ktv7t" pod "773296c4-147e-48e7-876b-f07250adb3f0" (UID: "773296c4-147e-48e7-876b-f07250adb3f0"). InnerVolumeSpecName "kube-api-access-ktv7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:03.548044 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:03.548006 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/773296c4-147e-48e7-876b-f07250adb3f0-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:03.548044 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:03.548037 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktv7t\" (UniqueName: \"kubernetes.io/projected/773296c4-147e-48e7-876b-f07250adb3f0-kube-api-access-ktv7t\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:04.220402 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.220369 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" Oct 31 03:47:04.220402 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.220388 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h" event={"ID":"773296c4-147e-48e7-876b-f07250adb3f0","Type":"ContainerDied","Data":"c4123d7ff4efe739f1533a7c5f77ce1b25800bcabc12f5b1c7755f7b5afe231c"} Oct 31 03:47:04.220635 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.220447 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4123d7ff4efe739f1533a7c5f77ce1b25800bcabc12f5b1c7755f7b5afe231c" Oct 31 03:47:04.535723 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.535632 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm"] Oct 31 03:47:04.536281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.536263 4426 cpu_manager.go:414] "containerMap: removing container" podUID="773296c4-147e-48e7-876b-f07250adb3f0" containerName="revoke-keys" Oct 31 03:47:04.536281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.536280 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773296c4-147e-48e7-876b-f07250adb3f0" containerName="revoke-keys" Oct 31 03:47:04.536423 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.536364 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="773296c4-147e-48e7-876b-f07250adb3f0" containerName="revoke-keys" Oct 31 03:47:04.540149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.540110 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:04.594404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.594365 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:47:04.594404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.594369 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:47:04.598422 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.598395 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm"] Oct 31 03:47:04.639842 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.639797 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:47:04.640018 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.639951 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:47:04.662405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.661834 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6a8ee93-8hhmm\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:04.662405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.661947 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6a8ee93-8hhmm\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:04.662405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.662073 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmjzf\" (UniqueName: \"kubernetes.io/projected/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-kube-api-access-hmjzf\") pod \"maas-api-revoke-sub-c6a8ee93-8hhmm\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:04.763760 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.763719 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmjzf\" (UniqueName: \"kubernetes.io/projected/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-kube-api-access-hmjzf\") pod \"maas-api-revoke-sub-c6a8ee93-8hhmm\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:04.763972 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.763841 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6a8ee93-8hhmm\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:04.763972 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.763928 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6a8ee93-8hhmm\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:04.764565 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.764540 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6a8ee93-8hhmm\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:04.766322 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.766299 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6a8ee93-8hhmm\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:04.822664 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.822579 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmjzf\" (UniqueName: \"kubernetes.io/projected/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-kube-api-access-hmjzf\") pod \"maas-api-revoke-sub-c6a8ee93-8hhmm\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:04.855276 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:04.855239 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:05.233028 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.232998 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm"] Oct 31 03:47:05.689189 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.689102 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6"] Oct 31 03:47:05.693304 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.693280 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.742725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.742692 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-68aa3e-openshift-default-dockercfg-jktfb\"" Oct 31 03:47:05.775738 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.775655 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istiod-ca-cert\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.775738 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.775692 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-certs\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.775738 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.775721 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-data\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.775984 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.775868 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-podinfo\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.775984 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.775920 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-credential-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.776064 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.776026 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.776064 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.776052 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-envoy\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.776174 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.776077 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-token\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.776251 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.776224 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnhvq\" (UniqueName: \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-kube-api-access-jnhvq\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.792679 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.792645 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6"] Oct 31 03:47:05.877013 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.876976 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-podinfo\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877042 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-credential-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877097 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877151 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-envoy\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877190 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-token\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877241 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnhvq\" (UniqueName: \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-kube-api-access-jnhvq\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877564 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877319 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istiod-ca-cert\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877564 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877345 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-certs\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877564 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877372 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-data\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877717 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877595 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-credential-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877717 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877622 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877717 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877656 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-data\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.877894 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.877806 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-certs\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.878189 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.878165 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istiod-ca-cert\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.880067 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.880045 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-envoy\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.880529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.880509 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-podinfo\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.935590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.935556 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnhvq\" (UniqueName: \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-kube-api-access-jnhvq\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:05.935789 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:05.935765 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-token\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:06.006750 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:06.006712 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:06.241755 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:06.241723 4426 generic.go:356] "Generic (PLEG): container finished" podID="7a6fdbf1-87a8-42f9-8877-f4a5032bd585" containerID="530bdea53a6c57efe59e4becce555146719607fa1baac2be4a268c329e93e01b" exitCode=0 Oct 31 03:47:06.241934 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:06.241810 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" event={"ID":"7a6fdbf1-87a8-42f9-8877-f4a5032bd585","Type":"ContainerDied","Data":"530bdea53a6c57efe59e4becce555146719607fa1baac2be4a268c329e93e01b"} Oct 31 03:47:06.241934 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:06.241860 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" event={"ID":"7a6fdbf1-87a8-42f9-8877-f4a5032bd585","Type":"ContainerStarted","Data":"370c26d28357c9bee06ed179855ff78f2c745d1ef3fcfeae5986a4ea63e61626"} Oct 31 03:47:06.490626 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:06.490595 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6"] Oct 31 03:47:06.491474 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:47:06.491440 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb4b041_8a0b_4e29_a276_a9ee7039a581.slice/crio-b7eb03b16e3e58fb384fa1bf390e1c487f0844f0750aad15f39c349c847e1f16 WatchSource:0}: Error finding container b7eb03b16e3e58fb384fa1bf390e1c487f0844f0750aad15f39c349c847e1f16: Status 404 returned error can't find the container with id b7eb03b16e3e58fb384fa1bf390e1c487f0844f0750aad15f39c349c847e1f16 Oct 31 03:47:06.494598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:06.494566 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:47:06.494683 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:06.494632 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:47:06.494683 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:06.494660 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:47:06.813541 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:06.813506 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:47:07.253142 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.253083 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" event={"ID":"3bb4b041-8a0b-4e29-a276-a9ee7039a581","Type":"ContainerStarted","Data":"808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7"} Oct 31 03:47:07.253328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.253156 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" event={"ID":"3bb4b041-8a0b-4e29-a276-a9ee7039a581","Type":"ContainerStarted","Data":"b7eb03b16e3e58fb384fa1bf390e1c487f0844f0750aad15f39c349c847e1f16"} Oct 31 03:47:07.474799 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.474766 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:07.599483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.595609 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" podStartSLOduration=2.59558524 podStartE2EDuration="2.59558524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:07.370495532 +0000 UTC m=+1124.371120762" watchObservedRunningTime="2026-10-31 03:47:07.59558524 +0000 UTC m=+1124.596210468" Oct 31 03:47:07.606087 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.604538 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-openshift-ca\") pod \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " Oct 31 03:47:07.606087 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.604652 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-kube-api-access-hmjzf\" (UniqueName: \"kubernetes.io/projected/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-kube-api-access-hmjzf\") pod \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " Oct 31 03:47:07.606087 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.604714 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-xks-ca\") pod \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\" (UID: \"7a6fdbf1-87a8-42f9-8877-f4a5032bd585\") " Oct 31 03:47:07.607088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.607053 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-openshift-ca" pod "7a6fdbf1-87a8-42f9-8877-f4a5032bd585" (UID: "7a6fdbf1-87a8-42f9-8877-f4a5032bd585"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:07.609367 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.609336 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-xks-ca" pod "7a6fdbf1-87a8-42f9-8877-f4a5032bd585" (UID: "7a6fdbf1-87a8-42f9-8877-f4a5032bd585"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:07.610166 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.610141 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-kube-api-access-hmjzf" pod "7a6fdbf1-87a8-42f9-8877-f4a5032bd585" (UID: "7a6fdbf1-87a8-42f9-8877-f4a5032bd585"). InnerVolumeSpecName "kube-api-access-hmjzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:07.712148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.709903 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:07.712148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.709940 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmjzf\" (UniqueName: \"kubernetes.io/projected/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-kube-api-access-hmjzf\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:07.712148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:07.709959 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6fdbf1-87a8-42f9-8877-f4a5032bd585-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:08.007318 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:08.007267 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:08.009144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:08.009095 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:47:08.009303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:08.009176 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:47:08.266780 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:08.266674 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" event={"ID":"7a6fdbf1-87a8-42f9-8877-f4a5032bd585","Type":"ContainerDied","Data":"370c26d28357c9bee06ed179855ff78f2c745d1ef3fcfeae5986a4ea63e61626"} Oct 31 03:47:08.267100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:08.267065 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="370c26d28357c9bee06ed179855ff78f2c745d1ef3fcfeae5986a4ea63e61626" Oct 31 03:47:08.267281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:08.266692 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm" Oct 31 03:47:09.009161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:09.008577 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:47:09.009161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:09.008640 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:47:10.007570 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:10.007535 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:47:10.007752 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:10.007597 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:47:11.007046 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:11.007014 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:47:11.007654 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:11.007075 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:47:11.134150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:11.121778 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:11.155664 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:11.155397 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:11.300724 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:11.300636 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:47:12.007303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:12.007263 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:47:12.007727 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:12.007330 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:47:13.007441 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:13.007398 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:47:13.007851 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:13.007486 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:47:14.273776 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:14.273745 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:14.274323 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:14.274300 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:14.275393 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:14.275376 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 03:47:15.268796 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.268761 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss"] Oct 31 03:47:15.269353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.269325 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7a6fdbf1-87a8-42f9-8877-f4a5032bd585" containerName="revoke-keys" Oct 31 03:47:15.269353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.269344 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a6fdbf1-87a8-42f9-8877-f4a5032bd585" containerName="revoke-keys" Oct 31 03:47:15.269560 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.269527 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a6fdbf1-87a8-42f9-8877-f4a5032bd585" containerName="revoke-keys" Oct 31 03:47:15.273116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.273093 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:15.328684 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.328656 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:47:15.329109 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.328755 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:47:15.331452 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.331422 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8vdz\" (UniqueName: \"kubernetes.io/projected/1a545899-a391-4432-9587-6ac10f7e55a0-kube-api-access-z8vdz\") pod \"maas-api-revoke-sub-d31a7f25-8pdss\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:15.331680 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.331659 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31a7f25-8pdss\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:15.332094 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.331851 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31a7f25-8pdss\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:15.334916 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.334886 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss"] Oct 31 03:47:15.375022 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.374993 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:47:15.375262 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.375244 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:47:15.433678 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.433637 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31a7f25-8pdss\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:15.433885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.433714 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vdz\" (UniqueName: \"kubernetes.io/projected/1a545899-a391-4432-9587-6ac10f7e55a0-kube-api-access-z8vdz\") pod \"maas-api-revoke-sub-d31a7f25-8pdss\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:15.433885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.433781 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31a7f25-8pdss\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:15.434627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.434588 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31a7f25-8pdss\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:15.435961 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.435935 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31a7f25-8pdss\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:15.491545 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.491509 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vdz\" (UniqueName: \"kubernetes.io/projected/1a545899-a391-4432-9587-6ac10f7e55a0-kube-api-access-z8vdz\") pod \"maas-api-revoke-sub-d31a7f25-8pdss\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:15.586726 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.586645 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:15.965372 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:15.965341 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss"] Oct 31 03:47:15.966043 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:47:15.966014 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a545899_a391_4432_9587_6ac10f7e55a0.slice/crio-e311d30990191b7a3318306682ff2daaf2ef504c40a8dc29b8500bf3f0b0488f WatchSource:0}: Error finding container e311d30990191b7a3318306682ff2daaf2ef504c40a8dc29b8500bf3f0b0488f: Status 404 returned error can't find the container with id e311d30990191b7a3318306682ff2daaf2ef504c40a8dc29b8500bf3f0b0488f Oct 31 03:47:16.336540 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:16.336487 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" event={"ID":"1a545899-a391-4432-9587-6ac10f7e55a0","Type":"ContainerStarted","Data":"ee851c95732e5317e880b35422639db8d1b231a398d59a40f01ce9275cc6784b"} Oct 31 03:47:16.336969 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:16.336571 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" event={"ID":"1a545899-a391-4432-9587-6ac10f7e55a0","Type":"ContainerStarted","Data":"e311d30990191b7a3318306682ff2daaf2ef504c40a8dc29b8500bf3f0b0488f"} Oct 31 03:47:16.452409 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:16.452343 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" podStartSLOduration=1.452324617 podStartE2EDuration="1.452324617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:16.450024141 +0000 UTC m=+1133.450649370" watchObservedRunningTime="2026-10-31 03:47:16.452324617 +0000 UTC m=+1133.452949847" Oct 31 03:47:17.344925 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:17.344890 4426 generic.go:356] "Generic (PLEG): container finished" podID="1a545899-a391-4432-9587-6ac10f7e55a0" containerID="ee851c95732e5317e880b35422639db8d1b231a398d59a40f01ce9275cc6784b" exitCode=0 Oct 31 03:47:17.345343 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:17.344968 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" event={"ID":"1a545899-a391-4432-9587-6ac10f7e55a0","Type":"ContainerDied","Data":"ee851c95732e5317e880b35422639db8d1b231a398d59a40f01ce9275cc6784b"} Oct 31 03:47:18.508720 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:18.508692 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:18.573342 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:18.572191 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a545899-a391-4432-9587-6ac10f7e55a0-kube-api-access-z8vdz\" (UniqueName: \"kubernetes.io/projected/1a545899-a391-4432-9587-6ac10f7e55a0-kube-api-access-z8vdz\") pod \"1a545899-a391-4432-9587-6ac10f7e55a0\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " Oct 31 03:47:18.573342 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:18.572271 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-openshift-ca\") pod \"1a545899-a391-4432-9587-6ac10f7e55a0\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " Oct 31 03:47:18.573342 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:18.572311 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-xks-ca\") pod \"1a545899-a391-4432-9587-6ac10f7e55a0\" (UID: \"1a545899-a391-4432-9587-6ac10f7e55a0\") " Oct 31 03:47:18.573657 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:18.573419 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-openshift-ca" pod "1a545899-a391-4432-9587-6ac10f7e55a0" (UID: "1a545899-a391-4432-9587-6ac10f7e55a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:18.575300 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:18.575275 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-xks-ca" pod "1a545899-a391-4432-9587-6ac10f7e55a0" (UID: "1a545899-a391-4432-9587-6ac10f7e55a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:18.575862 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:18.575841 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a545899-a391-4432-9587-6ac10f7e55a0-kube-api-access-z8vdz" pod "1a545899-a391-4432-9587-6ac10f7e55a0" (UID: "1a545899-a391-4432-9587-6ac10f7e55a0"). InnerVolumeSpecName "kube-api-access-z8vdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:18.673721 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:18.673690 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8vdz\" (UniqueName: \"kubernetes.io/projected/1a545899-a391-4432-9587-6ac10f7e55a0-kube-api-access-z8vdz\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:18.673721 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:18.673719 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:18.673721 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:18.673729 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a545899-a391-4432-9587-6ac10f7e55a0-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.361165 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:19.361134 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" Oct 31 03:47:19.361363 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:19.361159 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss" event={"ID":"1a545899-a391-4432-9587-6ac10f7e55a0","Type":"ContainerDied","Data":"e311d30990191b7a3318306682ff2daaf2ef504c40a8dc29b8500bf3f0b0488f"} Oct 31 03:47:19.361363 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:19.361214 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e311d30990191b7a3318306682ff2daaf2ef504c40a8dc29b8500bf3f0b0488f" Oct 31 03:47:21.588723 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.588690 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v"] Oct 31 03:47:21.589302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.589287 4426 cpu_manager.go:414] "containerMap: removing container" podUID="1a545899-a391-4432-9587-6ac10f7e55a0" containerName="revoke-keys" Oct 31 03:47:21.589302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.589303 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a545899-a391-4432-9587-6ac10f7e55a0" containerName="revoke-keys" Oct 31 03:47:21.589405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.589392 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a545899-a391-4432-9587-6ac10f7e55a0" containerName="revoke-keys" Oct 31 03:47:21.593191 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.593170 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.659066 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.647190 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e5b8f9-openshift-default-dockercfg-qt6w5\"" Oct 31 03:47:21.699426 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.699393 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v"] Oct 31 03:47:21.709578 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.709537 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvpss\" (UniqueName: \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-kube-api-access-cvpss\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.709761 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.709653 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.709761 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.709716 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.709761 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.709750 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.709918 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.709796 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4614978e-00e9-4f01-9589-af551a28666f-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.709918 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.709830 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.709918 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.709875 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.710074 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.709952 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.710074 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.709989 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4614978e-00e9-4f01-9589-af551a28666f-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.811253 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.811210 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvpss\" (UniqueName: \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-kube-api-access-cvpss\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.811441 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.811323 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.811441 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.811391 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.811527 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.811435 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.811527 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.811485 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4614978e-00e9-4f01-9589-af551a28666f-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.811611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.811525 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.811611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.811571 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.811704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.811657 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.811704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.811694 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4614978e-00e9-4f01-9589-af551a28666f-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.811911 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.811885 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.812002 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.811980 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.812169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.812120 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.812169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.812150 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.812301 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.812266 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4614978e-00e9-4f01-9589-af551a28666f-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.813928 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.813905 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.814367 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.814349 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4614978e-00e9-4f01-9589-af551a28666f-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.870213 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.869728 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.870213 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.869954 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvpss\" (UniqueName: \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-kube-api-access-cvpss\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:21.907379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:21.907346 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:22.350422 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:22.350396 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v"] Oct 31 03:47:22.351254 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:47:22.351231 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4614978e_00e9_4f01_9589_af551a28666f.slice/crio-71b26e7fbed1f1fc9435d3b2cd06e0d69ad726d72f617535fef86428699c47e9 WatchSource:0}: Error finding container 71b26e7fbed1f1fc9435d3b2cd06e0d69ad726d72f617535fef86428699c47e9: Status 404 returned error can't find the container with id 71b26e7fbed1f1fc9435d3b2cd06e0d69ad726d72f617535fef86428699c47e9 Oct 31 03:47:22.353798 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:22.353762 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:47:22.353923 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:22.353854 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:47:22.353923 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:22.353905 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:47:22.383596 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:22.383564 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" event={"ID":"4614978e-00e9-4f01-9589-af551a28666f","Type":"ContainerStarted","Data":"71b26e7fbed1f1fc9435d3b2cd06e0d69ad726d72f617535fef86428699c47e9"} Oct 31 03:47:23.391932 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:23.391893 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" event={"ID":"4614978e-00e9-4f01-9589-af551a28666f","Type":"ContainerStarted","Data":"2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5"} Oct 31 03:47:23.555425 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:23.555372 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" podStartSLOduration=2.555357272 podStartE2EDuration="2.555357272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:23.552177106 +0000 UTC m=+1140.552802336" watchObservedRunningTime="2026-10-31 03:47:23.555357272 +0000 UTC m=+1140.555982502" Oct 31 03:47:23.907771 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:23.907737 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:23.909755 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:23.909722 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:47:23.909898 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:23.909784 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" podUID="4614978e-00e9-4f01-9589-af551a28666f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:47:24.908818 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:24.908778 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:47:24.909247 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:24.908846 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" podUID="4614978e-00e9-4f01-9589-af551a28666f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:47:25.908585 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:25.908549 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:47:25.908741 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:25.908615 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" podUID="4614978e-00e9-4f01-9589-af551a28666f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:47:26.908239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:26.908198 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:47:26.908634 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:26.908253 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" podUID="4614978e-00e9-4f01-9589-af551a28666f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:47:27.908069 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:27.908033 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:47:27.908242 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:27.908091 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" podUID="4614978e-00e9-4f01-9589-af551a28666f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:47:28.908718 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:28.908675 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:47:28.909117 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:28.908750 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" podUID="4614978e-00e9-4f01-9589-af551a28666f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:47:29.912490 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:29.912455 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:29.912970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:29.912949 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:29.913699 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:29.913679 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 03:47:32.849449 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:47:32.849410 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:47:33.022148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.019326 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km"] Oct 31 03:47:33.038146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.035711 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:33.081940 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.081896 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74547e64-zp7km\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:33.082207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.082187 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4kc\" (UniqueName: \"kubernetes.io/projected/63d912c1-83b4-448f-bf05-53fd3496625b-kube-api-access-mz4kc\") pod \"maas-api-revoke-sub-74547e64-zp7km\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:33.082410 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.082391 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74547e64-zp7km\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:33.116265 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.088805 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km"] Oct 31 03:47:33.138148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.123207 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:47:33.138148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.123333 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:47:33.138148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.123959 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:47:33.233153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.211787 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4kc\" (UniqueName: \"kubernetes.io/projected/63d912c1-83b4-448f-bf05-53fd3496625b-kube-api-access-mz4kc\") pod \"maas-api-revoke-sub-74547e64-zp7km\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:33.254167 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.233708 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74547e64-zp7km\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:33.257610 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.257579 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:47:33.267179 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.267109 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74547e64-zp7km\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:33.268310 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.268280 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74547e64-zp7km\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:33.347192 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.341572 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4kc\" (UniqueName: \"kubernetes.io/projected/63d912c1-83b4-448f-bf05-53fd3496625b-kube-api-access-mz4kc\") pod \"maas-api-revoke-sub-74547e64-zp7km\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:33.426360 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.426008 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74547e64-zp7km\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:33.682917 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:33.682411 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:34.147170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:34.139060 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km"] Oct 31 03:47:34.154277 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:47:34.154222 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d912c1_83b4_448f_bf05_53fd3496625b.slice/crio-09436cc5814913baac05f68349b72c714f32fbdab0cb670445084510eddea808 WatchSource:0}: Error finding container 09436cc5814913baac05f68349b72c714f32fbdab0cb670445084510eddea808: Status 404 returned error can't find the container with id 09436cc5814913baac05f68349b72c714f32fbdab0cb670445084510eddea808 Oct 31 03:47:34.504088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:34.504034 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" event={"ID":"63d912c1-83b4-448f-bf05-53fd3496625b","Type":"ContainerStarted","Data":"09436cc5814913baac05f68349b72c714f32fbdab0cb670445084510eddea808"} Oct 31 03:47:35.519190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:35.514888 4426 generic.go:356] "Generic (PLEG): container finished" podID="63d912c1-83b4-448f-bf05-53fd3496625b" containerID="4cb5ef8a3eb36d4a9dda21e61e1106d056f4394fa1dfc5159b4feb75794356f3" exitCode=0 Oct 31 03:47:35.519190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:35.514957 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" event={"ID":"63d912c1-83b4-448f-bf05-53fd3496625b","Type":"ContainerDied","Data":"4cb5ef8a3eb36d4a9dda21e61e1106d056f4394fa1dfc5159b4feb75794356f3"} Oct 31 03:47:36.688147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:36.688105 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:36.822200 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:36.820467 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d912c1-83b4-448f-bf05-53fd3496625b-kube-api-access-mz4kc\" (UniqueName: \"kubernetes.io/projected/63d912c1-83b4-448f-bf05-53fd3496625b-kube-api-access-mz4kc\") pod \"63d912c1-83b4-448f-bf05-53fd3496625b\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " Oct 31 03:47:36.822200 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:36.820532 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-xks-ca\") pod \"63d912c1-83b4-448f-bf05-53fd3496625b\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " Oct 31 03:47:36.822200 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:36.820738 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-openshift-ca\") pod \"63d912c1-83b4-448f-bf05-53fd3496625b\" (UID: \"63d912c1-83b4-448f-bf05-53fd3496625b\") " Oct 31 03:47:36.822200 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:36.821260 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-openshift-ca" pod "63d912c1-83b4-448f-bf05-53fd3496625b" (UID: "63d912c1-83b4-448f-bf05-53fd3496625b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:36.822200 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:36.821479 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:36.823489 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:36.823464 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-xks-ca" pod "63d912c1-83b4-448f-bf05-53fd3496625b" (UID: "63d912c1-83b4-448f-bf05-53fd3496625b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:36.823715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:36.823694 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d912c1-83b4-448f-bf05-53fd3496625b-kube-api-access-mz4kc" pod "63d912c1-83b4-448f-bf05-53fd3496625b" (UID: "63d912c1-83b4-448f-bf05-53fd3496625b"). InnerVolumeSpecName "kube-api-access-mz4kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:36.922990 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:36.922952 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz4kc\" (UniqueName: \"kubernetes.io/projected/63d912c1-83b4-448f-bf05-53fd3496625b-kube-api-access-mz4kc\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:36.922990 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:36.922984 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d912c1-83b4-448f-bf05-53fd3496625b-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:37.531174 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:37.531060 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" Oct 31 03:47:37.531174 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:37.531075 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km" event={"ID":"63d912c1-83b4-448f-bf05-53fd3496625b","Type":"ContainerDied","Data":"09436cc5814913baac05f68349b72c714f32fbdab0cb670445084510eddea808"} Oct 31 03:47:37.531174 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:37.531142 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09436cc5814913baac05f68349b72c714f32fbdab0cb670445084510eddea808" Oct 31 03:47:45.444648 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.444615 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr"] Oct 31 03:47:45.445193 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.445178 4426 cpu_manager.go:414] "containerMap: removing container" podUID="63d912c1-83b4-448f-bf05-53fd3496625b" containerName="revoke-keys" Oct 31 03:47:45.445193 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.445193 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d912c1-83b4-448f-bf05-53fd3496625b" containerName="revoke-keys" Oct 31 03:47:45.445320 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.445310 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d912c1-83b4-448f-bf05-53fd3496625b" containerName="revoke-keys" Oct 31 03:47:45.448847 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.448828 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:45.498909 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.498875 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:47:45.525247 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.525214 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa38e661-p5vqr\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:45.525431 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.525266 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa38e661-p5vqr\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:45.525431 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.525286 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlqkh\" (UniqueName: \"kubernetes.io/projected/53339386-561a-4a09-a76c-480ee83cc23e-kube-api-access-xlqkh\") pod \"maas-api-revoke-sub-aa38e661-p5vqr\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:45.545299 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.545261 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:47:45.545299 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.545273 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:47:45.545579 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.545439 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:47:45.549605 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.549580 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr"] Oct 31 03:47:45.627193 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.627145 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa38e661-p5vqr\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:45.627398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.627218 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa38e661-p5vqr\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:45.627398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.627247 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlqkh\" (UniqueName: \"kubernetes.io/projected/53339386-561a-4a09-a76c-480ee83cc23e-kube-api-access-xlqkh\") pod \"maas-api-revoke-sub-aa38e661-p5vqr\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:45.627877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.627852 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa38e661-p5vqr\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:45.629614 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.629589 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa38e661-p5vqr\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:45.684583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.684548 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlqkh\" (UniqueName: \"kubernetes.io/projected/53339386-561a-4a09-a76c-480ee83cc23e-kube-api-access-xlqkh\") pod \"maas-api-revoke-sub-aa38e661-p5vqr\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:45.764687 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:45.764609 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:46.139332 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:46.139305 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr"] Oct 31 03:47:46.139734 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:47:46.139709 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53339386_561a_4a09_a76c_480ee83cc23e.slice/crio-dcb38f654ec850e772d87cbc4eb307a3c9605bf63ce90c994abcc5c249dbff63 WatchSource:0}: Error finding container dcb38f654ec850e772d87cbc4eb307a3c9605bf63ce90c994abcc5c249dbff63: Status 404 returned error can't find the container with id dcb38f654ec850e772d87cbc4eb307a3c9605bf63ce90c994abcc5c249dbff63 Oct 31 03:47:46.602385 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:46.602339 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" event={"ID":"53339386-561a-4a09-a76c-480ee83cc23e","Type":"ContainerStarted","Data":"3e3c416b9496659c72ebe32983796a5feba43284cae37ff61bfea06a6d945baa"} Oct 31 03:47:46.602840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:46.602409 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" event={"ID":"53339386-561a-4a09-a76c-480ee83cc23e","Type":"ContainerStarted","Data":"dcb38f654ec850e772d87cbc4eb307a3c9605bf63ce90c994abcc5c249dbff63"} Oct 31 03:47:46.712099 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:46.712045 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" podStartSLOduration=1.7120297880000002 podStartE2EDuration="1.712029788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:46.711080709 +0000 UTC m=+1163.711705942" watchObservedRunningTime="2026-10-31 03:47:46.712029788 +0000 UTC m=+1163.712655016" Oct 31 03:47:47.611488 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:47.611445 4426 generic.go:356] "Generic (PLEG): container finished" podID="53339386-561a-4a09-a76c-480ee83cc23e" containerID="3e3c416b9496659c72ebe32983796a5feba43284cae37ff61bfea06a6d945baa" exitCode=0 Oct 31 03:47:47.611890 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:47.611523 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" event={"ID":"53339386-561a-4a09-a76c-480ee83cc23e","Type":"ContainerDied","Data":"3e3c416b9496659c72ebe32983796a5feba43284cae37ff61bfea06a6d945baa"} Oct 31 03:47:48.775096 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:48.775066 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:48.862777 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:48.862743 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-openshift-ca\") pod \"53339386-561a-4a09-a76c-480ee83cc23e\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " Oct 31 03:47:48.862966 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:48.862945 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53339386-561a-4a09-a76c-480ee83cc23e-kube-api-access-xlqkh\" (UniqueName: \"kubernetes.io/projected/53339386-561a-4a09-a76c-480ee83cc23e-kube-api-access-xlqkh\") pod \"53339386-561a-4a09-a76c-480ee83cc23e\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " Oct 31 03:47:48.863179 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:48.863119 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-xks-ca\") pod \"53339386-561a-4a09-a76c-480ee83cc23e\" (UID: \"53339386-561a-4a09-a76c-480ee83cc23e\") " Oct 31 03:47:48.863179 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:48.863170 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-openshift-ca" pod "53339386-561a-4a09-a76c-480ee83cc23e" (UID: "53339386-561a-4a09-a76c-480ee83cc23e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:48.863512 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:48.863491 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:48.865085 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:48.865049 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-xks-ca" pod "53339386-561a-4a09-a76c-480ee83cc23e" (UID: "53339386-561a-4a09-a76c-480ee83cc23e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:48.865263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:48.865108 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53339386-561a-4a09-a76c-480ee83cc23e-kube-api-access-xlqkh" pod "53339386-561a-4a09-a76c-480ee83cc23e" (UID: "53339386-561a-4a09-a76c-480ee83cc23e"). InnerVolumeSpecName "kube-api-access-xlqkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:48.964958 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:48.964923 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlqkh\" (UniqueName: \"kubernetes.io/projected/53339386-561a-4a09-a76c-480ee83cc23e-kube-api-access-xlqkh\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:48.964958 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:48.964957 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53339386-561a-4a09-a76c-480ee83cc23e-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:49.627390 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:49.627355 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" Oct 31 03:47:49.627390 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:49.627380 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr" event={"ID":"53339386-561a-4a09-a76c-480ee83cc23e","Type":"ContainerDied","Data":"dcb38f654ec850e772d87cbc4eb307a3c9605bf63ce90c994abcc5c249dbff63"} Oct 31 03:47:49.627614 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:49.627437 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcb38f654ec850e772d87cbc4eb307a3c9605bf63ce90c994abcc5c249dbff63" Oct 31 03:47:54.611155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.611102 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w"] Oct 31 03:47:54.611699 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.611685 4426 cpu_manager.go:414] "containerMap: removing container" podUID="53339386-561a-4a09-a76c-480ee83cc23e" containerName="revoke-keys" Oct 31 03:47:54.611743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.611699 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53339386-561a-4a09-a76c-480ee83cc23e" containerName="revoke-keys" Oct 31 03:47:54.611789 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.611780 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="53339386-561a-4a09-a76c-480ee83cc23e" containerName="revoke-keys" Oct 31 03:47:54.615468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.615446 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:54.665180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.665151 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:47:54.671014 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.670980 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w"] Oct 31 03:47:54.711784 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.711744 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:47:54.711784 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.711750 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:47:54.711784 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.711750 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:47:54.736421 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.736379 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704dcc4a-cpx8w\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:54.736600 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.736447 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krqw8\" (UniqueName: \"kubernetes.io/projected/aeb1caa6-9b1d-45de-912d-83b7af928e4d-kube-api-access-krqw8\") pod \"maas-api-revoke-sub-704dcc4a-cpx8w\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:54.736600 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.736576 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704dcc4a-cpx8w\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:54.837509 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.837468 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704dcc4a-cpx8w\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:54.837729 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.837525 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krqw8\" (UniqueName: \"kubernetes.io/projected/aeb1caa6-9b1d-45de-912d-83b7af928e4d-kube-api-access-krqw8\") pod \"maas-api-revoke-sub-704dcc4a-cpx8w\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:54.837729 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.837586 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704dcc4a-cpx8w\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:54.838400 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.838371 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704dcc4a-cpx8w\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:54.839948 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.839926 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704dcc4a-cpx8w\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:54.898612 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.898581 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krqw8\" (UniqueName: \"kubernetes.io/projected/aeb1caa6-9b1d-45de-912d-83b7af928e4d-kube-api-access-krqw8\") pod \"maas-api-revoke-sub-704dcc4a-cpx8w\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:54.932213 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:54.932160 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:55.348008 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.345562 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w"] Oct 31 03:47:55.349141 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:47:55.348705 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb1caa6_9b1d_45de_912d_83b7af928e4d.slice/crio-356c1e4b26bf3bdb29c68cb2235a82c4c0b5d7b98f3c6b0d900288173c619c1c WatchSource:0}: Error finding container 356c1e4b26bf3bdb29c68cb2235a82c4c0b5d7b98f3c6b0d900288173c619c1c: Status 404 returned error can't find the container with id 356c1e4b26bf3bdb29c68cb2235a82c4c0b5d7b98f3c6b0d900288173c619c1c Oct 31 03:47:55.544743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.544705 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n"] Oct 31 03:47:55.548941 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.548915 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:55.607199 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.606498 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n"] Oct 31 03:47:55.648180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.648134 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383900a6-vz24n\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:55.648568 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.648189 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383900a6-vz24n\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:55.648568 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.648234 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kx82\" (UniqueName: \"kubernetes.io/projected/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-kube-api-access-9kx82\") pod \"maas-api-revoke-sub-383900a6-vz24n\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:55.700866 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.700824 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" event={"ID":"aeb1caa6-9b1d-45de-912d-83b7af928e4d","Type":"ContainerStarted","Data":"95a7ccd196bc168ea3389fd07aad680b91f97bbb529e2c55e8d146196026c17c"} Oct 31 03:47:55.701235 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.701200 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" event={"ID":"aeb1caa6-9b1d-45de-912d-83b7af928e4d","Type":"ContainerStarted","Data":"356c1e4b26bf3bdb29c68cb2235a82c4c0b5d7b98f3c6b0d900288173c619c1c"} Oct 31 03:47:55.750074 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.750025 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383900a6-vz24n\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:55.750271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.750098 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383900a6-vz24n\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:55.750271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.750202 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kx82\" (UniqueName: \"kubernetes.io/projected/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-kube-api-access-9kx82\") pod \"maas-api-revoke-sub-383900a6-vz24n\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:55.750935 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.750907 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383900a6-vz24n\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:55.752667 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.752643 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383900a6-vz24n\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:55.831932 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.831869 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" podStartSLOduration=1.831850116 podStartE2EDuration="1.831850116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:55.827601663 +0000 UTC m=+1172.828226891" watchObservedRunningTime="2026-10-31 03:47:55.831850116 +0000 UTC m=+1172.832475346" Oct 31 03:47:55.835364 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.835332 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kx82\" (UniqueName: \"kubernetes.io/projected/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-kube-api-access-9kx82\") pod \"maas-api-revoke-sub-383900a6-vz24n\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:55.864795 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.864712 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:55.972692 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:55.972660 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:47:56.250627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:56.250596 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n"] Oct 31 03:47:56.251571 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:47:56.251539 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9f6e0f_dfd5_48a5_8b23_c921247f0bd5.slice/crio-798cac3134e2e855eabd7fce21893340f7fc4957f50503f4d22cadec6eea334f WatchSource:0}: Error finding container 798cac3134e2e855eabd7fce21893340f7fc4957f50503f4d22cadec6eea334f: Status 404 returned error can't find the container with id 798cac3134e2e855eabd7fce21893340f7fc4957f50503f4d22cadec6eea334f Oct 31 03:47:56.722460 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:56.722417 4426 generic.go:356] "Generic (PLEG): container finished" podID="aeb1caa6-9b1d-45de-912d-83b7af928e4d" containerID="95a7ccd196bc168ea3389fd07aad680b91f97bbb529e2c55e8d146196026c17c" exitCode=0 Oct 31 03:47:56.722924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:56.722875 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" event={"ID":"aeb1caa6-9b1d-45de-912d-83b7af928e4d","Type":"ContainerDied","Data":"95a7ccd196bc168ea3389fd07aad680b91f97bbb529e2c55e8d146196026c17c"} Oct 31 03:47:56.732922 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:56.732162 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" event={"ID":"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5","Type":"ContainerStarted","Data":"e1fbd8925760e13d4c253d55578cafeeaca621c2356fad3bf3cdf736d58dffd9"} Oct 31 03:47:56.732922 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:56.732230 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" event={"ID":"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5","Type":"ContainerStarted","Data":"798cac3134e2e855eabd7fce21893340f7fc4957f50503f4d22cadec6eea334f"} Oct 31 03:47:57.010150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:56.988233 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" podStartSLOduration=1.988210104 podStartE2EDuration="1.988210104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:56.946652037 +0000 UTC m=+1173.947277268" watchObservedRunningTime="2026-10-31 03:47:56.988210104 +0000 UTC m=+1173.988835335" Oct 31 03:47:57.382149 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:47:57.381521 4426 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9f6e0f_dfd5_48a5_8b23_c921247f0bd5.slice/crio-conmon-e1fbd8925760e13d4c253d55578cafeeaca621c2356fad3bf3cdf736d58dffd9.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:47:57.753160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:57.752293 4426 generic.go:356] "Generic (PLEG): container finished" podID="8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5" containerID="e1fbd8925760e13d4c253d55578cafeeaca621c2356fad3bf3cdf736d58dffd9" exitCode=0 Oct 31 03:47:57.753160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:57.752868 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" event={"ID":"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5","Type":"ContainerDied","Data":"e1fbd8925760e13d4c253d55578cafeeaca621c2356fad3bf3cdf736d58dffd9"} Oct 31 03:47:58.018344 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.018260 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:58.107292 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.107254 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-openshift-ca\") pod \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " Oct 31 03:47:58.107471 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.107451 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-xks-ca\") pod \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " Oct 31 03:47:58.107540 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.107526 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb1caa6-9b1d-45de-912d-83b7af928e4d-kube-api-access-krqw8\" (UniqueName: \"kubernetes.io/projected/aeb1caa6-9b1d-45de-912d-83b7af928e4d-kube-api-access-krqw8\") pod \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\" (UID: \"aeb1caa6-9b1d-45de-912d-83b7af928e4d\") " Oct 31 03:47:58.107739 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.107706 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-openshift-ca" pod "aeb1caa6-9b1d-45de-912d-83b7af928e4d" (UID: "aeb1caa6-9b1d-45de-912d-83b7af928e4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:58.108035 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.108010 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.109579 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.109546 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-xks-ca" pod "aeb1caa6-9b1d-45de-912d-83b7af928e4d" (UID: "aeb1caa6-9b1d-45de-912d-83b7af928e4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:58.109672 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.109607 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb1caa6-9b1d-45de-912d-83b7af928e4d-kube-api-access-krqw8" pod "aeb1caa6-9b1d-45de-912d-83b7af928e4d" (UID: "aeb1caa6-9b1d-45de-912d-83b7af928e4d"). InnerVolumeSpecName "kube-api-access-krqw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:58.208790 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.208749 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb1caa6-9b1d-45de-912d-83b7af928e4d-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.208790 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.208780 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krqw8\" (UniqueName: \"kubernetes.io/projected/aeb1caa6-9b1d-45de-912d-83b7af928e4d-kube-api-access-krqw8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.762139 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.762106 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" Oct 31 03:47:58.762640 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.762143 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w" event={"ID":"aeb1caa6-9b1d-45de-912d-83b7af928e4d","Type":"ContainerDied","Data":"356c1e4b26bf3bdb29c68cb2235a82c4c0b5d7b98f3c6b0d900288173c619c1c"} Oct 31 03:47:58.762640 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.762200 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="356c1e4b26bf3bdb29c68cb2235a82c4c0b5d7b98f3c6b0d900288173c619c1c" Oct 31 03:47:58.916659 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:58.916629 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:59.019185 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.019064 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-xks-ca\") pod \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " Oct 31 03:47:59.019358 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.019257 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-kube-api-access-9kx82\" (UniqueName: \"kubernetes.io/projected/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-kube-api-access-9kx82\") pod \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " Oct 31 03:47:59.019430 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.019354 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-openshift-ca\") pod \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\" (UID: \"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5\") " Oct 31 03:47:59.019815 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.019783 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-openshift-ca" pod "8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5" (UID: "8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:59.021847 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.021808 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-kube-api-access-9kx82" pod "8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5" (UID: "8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5"). InnerVolumeSpecName "kube-api-access-9kx82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:59.022147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.022101 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-xks-ca" pod "8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5" (UID: "8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:59.121142 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.121084 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:59.121316 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.121148 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kx82\" (UniqueName: \"kubernetes.io/projected/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-kube-api-access-9kx82\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:59.121316 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.121164 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:59.770788 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.770754 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" Oct 31 03:47:59.771194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.770790 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n" event={"ID":"8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5","Type":"ContainerDied","Data":"798cac3134e2e855eabd7fce21893340f7fc4957f50503f4d22cadec6eea334f"} Oct 31 03:47:59.771194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:47:59.770844 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="798cac3134e2e855eabd7fce21893340f7fc4957f50503f4d22cadec6eea334f" Oct 31 03:48:02.849633 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:48:02.849587 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:48:10.973789 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:10.973742 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb"] Oct 31 03:48:10.974586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:10.974563 4426 cpu_manager.go:414] "containerMap: removing container" podUID="aeb1caa6-9b1d-45de-912d-83b7af928e4d" containerName="revoke-keys" Oct 31 03:48:10.974690 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:10.974588 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb1caa6-9b1d-45de-912d-83b7af928e4d" containerName="revoke-keys" Oct 31 03:48:10.974690 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:10.974608 4426 cpu_manager.go:414] "containerMap: removing container" podUID="8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5" containerName="revoke-keys" Oct 31 03:48:10.974690 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:10.974615 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5" containerName="revoke-keys" Oct 31 03:48:10.974845 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:10.974748 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb1caa6-9b1d-45de-912d-83b7af928e4d" containerName="revoke-keys" Oct 31 03:48:10.974845 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:10.974762 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5" containerName="revoke-keys" Oct 31 03:48:10.979420 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:10.979395 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:11.041411 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.041378 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb"] Oct 31 03:48:11.073600 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.073556 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b7447e7-hlxqb\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:11.073802 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.073637 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b7447e7-hlxqb\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:11.073802 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.073740 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6znp\" (UniqueName: \"kubernetes.io/projected/d528b635-9901-4bc4-8936-58997ca5b278-kube-api-access-g6znp\") pod \"maas-api-revoke-sub-7b7447e7-hlxqb\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:11.075247 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.075209 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:48:11.075460 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.075291 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:48:11.075559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.075330 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:48:11.075559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.075405 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:48:11.174505 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.174472 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b7447e7-hlxqb\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:11.174677 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.174618 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6znp\" (UniqueName: \"kubernetes.io/projected/d528b635-9901-4bc4-8936-58997ca5b278-kube-api-access-g6znp\") pod \"maas-api-revoke-sub-7b7447e7-hlxqb\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:11.174715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.174689 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b7447e7-hlxqb\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:11.175204 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.175163 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b7447e7-hlxqb\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:11.176961 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.176933 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b7447e7-hlxqb\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:11.203425 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.203386 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs"] Oct 31 03:48:11.207751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.207729 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:11.232233 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.232163 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6znp\" (UniqueName: \"kubernetes.io/projected/d528b635-9901-4bc4-8936-58997ca5b278-kube-api-access-g6znp\") pod \"maas-api-revoke-sub-7b7447e7-hlxqb\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:11.263221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.263185 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs"] Oct 31 03:48:11.293154 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.293087 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:11.376725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.376681 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d04429-nx5fs\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:11.376888 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.376813 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d04429-nx5fs\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:11.376990 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.376946 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgcxg\" (UniqueName: \"kubernetes.io/projected/68b11ee5-9421-4de2-8948-4a55e683ba1a-kube-api-access-jgcxg\") pod \"maas-api-revoke-sub-59d04429-nx5fs\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:11.477870 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.477834 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgcxg\" (UniqueName: \"kubernetes.io/projected/68b11ee5-9421-4de2-8948-4a55e683ba1a-kube-api-access-jgcxg\") pod \"maas-api-revoke-sub-59d04429-nx5fs\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:11.478058 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.477937 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d04429-nx5fs\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:11.478058 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.478031 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d04429-nx5fs\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:11.478792 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.478768 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d04429-nx5fs\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:11.480393 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.480370 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d04429-nx5fs\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:11.535600 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.535511 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgcxg\" (UniqueName: \"kubernetes.io/projected/68b11ee5-9421-4de2-8948-4a55e683ba1a-kube-api-access-jgcxg\") pod \"maas-api-revoke-sub-59d04429-nx5fs\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:11.679269 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:11.679237 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd528b635_9901_4bc4_8936_58997ca5b278.slice/crio-de1960a7fa11d6ccb31c775cb5a849089a38e9a83b64562a761f2105068b5c2d WatchSource:0}: Error finding container de1960a7fa11d6ccb31c775cb5a849089a38e9a83b64562a761f2105068b5c2d: Status 404 returned error can't find the container with id de1960a7fa11d6ccb31c775cb5a849089a38e9a83b64562a761f2105068b5c2d Oct 31 03:48:11.680204 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.680180 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb"] Oct 31 03:48:11.821949 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.821863 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:11.940057 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:11.938713 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" event={"ID":"d528b635-9901-4bc4-8936-58997ca5b278","Type":"ContainerStarted","Data":"de1960a7fa11d6ccb31c775cb5a849089a38e9a83b64562a761f2105068b5c2d"} Oct 31 03:48:12.287922 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.287893 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs"] Oct 31 03:48:12.288727 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:12.288700 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b11ee5_9421_4de2_8948_4a55e683ba1a.slice/crio-d0e2090d7170e441015c8704a768b4be7b37b514e54d41058a38792055ba416d WatchSource:0}: Error finding container d0e2090d7170e441015c8704a768b4be7b37b514e54d41058a38792055ba416d: Status 404 returned error can't find the container with id d0e2090d7170e441015c8704a768b4be7b37b514e54d41058a38792055ba416d Oct 31 03:48:12.730151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.729307 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg"] Oct 31 03:48:12.735331 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.735291 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:12.790682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.790648 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg"] Oct 31 03:48:12.801468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.801380 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a90f663-vjzqg\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:12.801468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.801438 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a90f663-vjzqg\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:12.801682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.801547 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldtlb\" (UniqueName: \"kubernetes.io/projected/ec8482bf-17f1-4a3d-b705-015efeb50fe9-kube-api-access-ldtlb\") pod \"maas-api-revoke-sub-7a90f663-vjzqg\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:12.902521 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.902486 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a90f663-vjzqg\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:12.902698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.902544 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a90f663-vjzqg\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:12.902698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.902586 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldtlb\" (UniqueName: \"kubernetes.io/projected/ec8482bf-17f1-4a3d-b705-015efeb50fe9-kube-api-access-ldtlb\") pod \"maas-api-revoke-sub-7a90f663-vjzqg\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:12.903158 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.903104 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a90f663-vjzqg\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:12.904966 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.904934 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a90f663-vjzqg\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:12.951243 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.951108 4426 generic.go:356] "Generic (PLEG): container finished" podID="d528b635-9901-4bc4-8936-58997ca5b278" containerID="04404620550be34553d1db7c941cecc2940e5296cf64015d0f29b959ec5a7c43" exitCode=0 Oct 31 03:48:12.951243 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.951159 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" event={"ID":"d528b635-9901-4bc4-8936-58997ca5b278","Type":"ContainerDied","Data":"04404620550be34553d1db7c941cecc2940e5296cf64015d0f29b959ec5a7c43"} Oct 31 03:48:12.953326 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.953295 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" event={"ID":"68b11ee5-9421-4de2-8948-4a55e683ba1a","Type":"ContainerStarted","Data":"5b3ff52feb81268cdbda876ad4f47e51df1d791676370b64f501f3b613a299c6"} Oct 31 03:48:12.953457 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.953356 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" event={"ID":"68b11ee5-9421-4de2-8948-4a55e683ba1a","Type":"ContainerStarted","Data":"d0e2090d7170e441015c8704a768b4be7b37b514e54d41058a38792055ba416d"} Oct 31 03:48:12.960048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:12.960026 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldtlb\" (UniqueName: \"kubernetes.io/projected/ec8482bf-17f1-4a3d-b705-015efeb50fe9-kube-api-access-ldtlb\") pod \"maas-api-revoke-sub-7a90f663-vjzqg\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:13.052376 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:13.052290 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:13.174606 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:13.174537 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" podStartSLOduration=2.174516755 podStartE2EDuration="2.174516755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:13.17353141 +0000 UTC m=+1190.174156650" watchObservedRunningTime="2026-10-31 03:48:13.174516755 +0000 UTC m=+1190.175141986" Oct 31 03:48:13.495304 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:13.495269 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8482bf_17f1_4a3d_b705_015efeb50fe9.slice/crio-8ef3f55a65074a9d0c84e7f9720abb97e37cf2952b64c3f957615064d5c61b7b WatchSource:0}: Error finding container 8ef3f55a65074a9d0c84e7f9720abb97e37cf2952b64c3f957615064d5c61b7b: Status 404 returned error can't find the container with id 8ef3f55a65074a9d0c84e7f9720abb97e37cf2952b64c3f957615064d5c61b7b Oct 31 03:48:13.495660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:13.495314 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg"] Oct 31 03:48:13.963019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:13.962974 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" event={"ID":"ec8482bf-17f1-4a3d-b705-015efeb50fe9","Type":"ContainerStarted","Data":"9094672e56c4a453cc478580bf0bd2b2a59441f0f8acea0918d81499831d292b"} Oct 31 03:48:13.963763 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:13.963044 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" event={"ID":"ec8482bf-17f1-4a3d-b705-015efeb50fe9","Type":"ContainerStarted","Data":"8ef3f55a65074a9d0c84e7f9720abb97e37cf2952b64c3f957615064d5c61b7b"} Oct 31 03:48:13.964886 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:13.964862 4426 generic.go:356] "Generic (PLEG): container finished" podID="68b11ee5-9421-4de2-8948-4a55e683ba1a" containerID="5b3ff52feb81268cdbda876ad4f47e51df1d791676370b64f501f3b613a299c6" exitCode=0 Oct 31 03:48:13.965014 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:13.964958 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" event={"ID":"68b11ee5-9421-4de2-8948-4a55e683ba1a","Type":"ContainerDied","Data":"5b3ff52feb81268cdbda876ad4f47e51df1d791676370b64f501f3b613a299c6"} Oct 31 03:48:14.076084 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.076027 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" podStartSLOduration=2.076011412 podStartE2EDuration="2.076011412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:14.071402969 +0000 UTC m=+1191.072028197" watchObservedRunningTime="2026-10-31 03:48:14.076011412 +0000 UTC m=+1191.076636640" Oct 31 03:48:14.128585 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.128550 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:14.223626 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.223594 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-xks-ca\") pod \"d528b635-9901-4bc4-8936-58997ca5b278\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " Oct 31 03:48:14.223820 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.223674 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d528b635-9901-4bc4-8936-58997ca5b278-kube-api-access-g6znp\" (UniqueName: \"kubernetes.io/projected/d528b635-9901-4bc4-8936-58997ca5b278-kube-api-access-g6znp\") pod \"d528b635-9901-4bc4-8936-58997ca5b278\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " Oct 31 03:48:14.223820 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.223703 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-openshift-ca\") pod \"d528b635-9901-4bc4-8936-58997ca5b278\" (UID: \"d528b635-9901-4bc4-8936-58997ca5b278\") " Oct 31 03:48:14.224183 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.224141 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-openshift-ca" pod "d528b635-9901-4bc4-8936-58997ca5b278" (UID: "d528b635-9901-4bc4-8936-58997ca5b278"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:14.226219 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.226192 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-xks-ca" pod "d528b635-9901-4bc4-8936-58997ca5b278" (UID: "d528b635-9901-4bc4-8936-58997ca5b278"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:14.226308 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.226245 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d528b635-9901-4bc4-8936-58997ca5b278-kube-api-access-g6znp" pod "d528b635-9901-4bc4-8936-58997ca5b278" (UID: "d528b635-9901-4bc4-8936-58997ca5b278"). InnerVolumeSpecName "kube-api-access-g6znp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:14.325623 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.325544 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:14.325623 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.325576 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6znp\" (UniqueName: \"kubernetes.io/projected/d528b635-9901-4bc4-8936-58997ca5b278-kube-api-access-g6znp\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:14.325623 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.325588 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d528b635-9901-4bc4-8936-58997ca5b278-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:14.973680 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.973648 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" Oct 31 03:48:14.974184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.973680 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb" event={"ID":"d528b635-9901-4bc4-8936-58997ca5b278","Type":"ContainerDied","Data":"de1960a7fa11d6ccb31c775cb5a849089a38e9a83b64562a761f2105068b5c2d"} Oct 31 03:48:14.974184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.973747 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de1960a7fa11d6ccb31c775cb5a849089a38e9a83b64562a761f2105068b5c2d" Oct 31 03:48:14.975399 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.975371 4426 generic.go:356] "Generic (PLEG): container finished" podID="ec8482bf-17f1-4a3d-b705-015efeb50fe9" containerID="9094672e56c4a453cc478580bf0bd2b2a59441f0f8acea0918d81499831d292b" exitCode=0 Oct 31 03:48:14.975510 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:14.975444 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" event={"ID":"ec8482bf-17f1-4a3d-b705-015efeb50fe9","Type":"ContainerDied","Data":"9094672e56c4a453cc478580bf0bd2b2a59441f0f8acea0918d81499831d292b"} Oct 31 03:48:15.141742 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.141712 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:15.234417 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.234324 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-xks-ca\") pod \"68b11ee5-9421-4de2-8948-4a55e683ba1a\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " Oct 31 03:48:15.234417 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.234370 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-openshift-ca\") pod \"68b11ee5-9421-4de2-8948-4a55e683ba1a\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " Oct 31 03:48:15.234640 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.234486 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b11ee5-9421-4de2-8948-4a55e683ba1a-kube-api-access-jgcxg\" (UniqueName: \"kubernetes.io/projected/68b11ee5-9421-4de2-8948-4a55e683ba1a-kube-api-access-jgcxg\") pod \"68b11ee5-9421-4de2-8948-4a55e683ba1a\" (UID: \"68b11ee5-9421-4de2-8948-4a55e683ba1a\") " Oct 31 03:48:15.234843 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.234814 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-openshift-ca" pod "68b11ee5-9421-4de2-8948-4a55e683ba1a" (UID: "68b11ee5-9421-4de2-8948-4a55e683ba1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:15.234982 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.234964 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:15.236592 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.236566 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b11ee5-9421-4de2-8948-4a55e683ba1a-kube-api-access-jgcxg" pod "68b11ee5-9421-4de2-8948-4a55e683ba1a" (UID: "68b11ee5-9421-4de2-8948-4a55e683ba1a"). InnerVolumeSpecName "kube-api-access-jgcxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:15.236592 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.236574 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-xks-ca" pod "68b11ee5-9421-4de2-8948-4a55e683ba1a" (UID: "68b11ee5-9421-4de2-8948-4a55e683ba1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:15.336340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.336298 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b11ee5-9421-4de2-8948-4a55e683ba1a-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:15.336340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.336336 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgcxg\" (UniqueName: \"kubernetes.io/projected/68b11ee5-9421-4de2-8948-4a55e683ba1a-kube-api-access-jgcxg\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:15.983815 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.983784 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" Oct 31 03:48:15.984348 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.983812 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs" event={"ID":"68b11ee5-9421-4de2-8948-4a55e683ba1a","Type":"ContainerDied","Data":"d0e2090d7170e441015c8704a768b4be7b37b514e54d41058a38792055ba416d"} Oct 31 03:48:15.984348 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:15.983857 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0e2090d7170e441015c8704a768b4be7b37b514e54d41058a38792055ba416d" Oct 31 03:48:16.135839 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.135815 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:16.146251 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.146212 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8482bf-17f1-4a3d-b705-015efeb50fe9-kube-api-access-ldtlb\" (UniqueName: \"kubernetes.io/projected/ec8482bf-17f1-4a3d-b705-015efeb50fe9-kube-api-access-ldtlb\") pod \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " Oct 31 03:48:16.146445 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.146269 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-openshift-ca\") pod \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " Oct 31 03:48:16.146522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.146486 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-xks-ca\") pod \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\" (UID: \"ec8482bf-17f1-4a3d-b705-015efeb50fe9\") " Oct 31 03:48:16.146931 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.146644 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-openshift-ca" pod "ec8482bf-17f1-4a3d-b705-015efeb50fe9" (UID: "ec8482bf-17f1-4a3d-b705-015efeb50fe9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:16.146931 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.146913 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:16.148749 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.148718 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8482bf-17f1-4a3d-b705-015efeb50fe9-kube-api-access-ldtlb" pod "ec8482bf-17f1-4a3d-b705-015efeb50fe9" (UID: "ec8482bf-17f1-4a3d-b705-015efeb50fe9"). InnerVolumeSpecName "kube-api-access-ldtlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:16.148965 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.148938 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-xks-ca" pod "ec8482bf-17f1-4a3d-b705-015efeb50fe9" (UID: "ec8482bf-17f1-4a3d-b705-015efeb50fe9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:16.248233 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.248079 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8482bf-17f1-4a3d-b705-015efeb50fe9-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:16.248233 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.248115 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldtlb\" (UniqueName: \"kubernetes.io/projected/ec8482bf-17f1-4a3d-b705-015efeb50fe9-kube-api-access-ldtlb\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:16.996896 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.996862 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" Oct 31 03:48:16.997332 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.996861 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg" event={"ID":"ec8482bf-17f1-4a3d-b705-015efeb50fe9","Type":"ContainerDied","Data":"8ef3f55a65074a9d0c84e7f9720abb97e37cf2952b64c3f957615064d5c61b7b"} Oct 31 03:48:16.997332 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:16.996985 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ef3f55a65074a9d0c84e7f9720abb97e37cf2952b64c3f957615064d5c61b7b" Oct 31 03:48:21.163348 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.163310 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7"] Oct 31 03:48:21.163912 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.163897 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ec8482bf-17f1-4a3d-b705-015efeb50fe9" containerName="revoke-keys" Oct 31 03:48:21.163963 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.163913 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8482bf-17f1-4a3d-b705-015efeb50fe9" containerName="revoke-keys" Oct 31 03:48:21.163963 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.163927 4426 cpu_manager.go:414] "containerMap: removing container" podUID="68b11ee5-9421-4de2-8948-4a55e683ba1a" containerName="revoke-keys" Oct 31 03:48:21.163963 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.163932 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b11ee5-9421-4de2-8948-4a55e683ba1a" containerName="revoke-keys" Oct 31 03:48:21.163963 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.163964 4426 cpu_manager.go:414] "containerMap: removing container" podUID="d528b635-9901-4bc4-8936-58997ca5b278" containerName="revoke-keys" Oct 31 03:48:21.164094 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.163969 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d528b635-9901-4bc4-8936-58997ca5b278" containerName="revoke-keys" Oct 31 03:48:21.164094 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.164055 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="d528b635-9901-4bc4-8936-58997ca5b278" containerName="revoke-keys" Oct 31 03:48:21.164094 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.164064 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8482bf-17f1-4a3d-b705-015efeb50fe9" containerName="revoke-keys" Oct 31 03:48:21.164094 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.164070 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b11ee5-9421-4de2-8948-4a55e683ba1a" containerName="revoke-keys" Oct 31 03:48:21.167894 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.167874 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:21.211015 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.210973 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wgh\" (UniqueName: \"kubernetes.io/projected/045b5abd-acde-4a56-8e6e-aecb2f7de383-kube-api-access-d5wgh\") pod \"maas-api-revoke-sub-08f0902d-8fqf7\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:21.211199 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.211028 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f0902d-8fqf7\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:21.211302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.211277 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f0902d-8fqf7\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:21.263462 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.263428 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:48:21.263462 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.263456 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:48:21.263693 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.263483 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:48:21.263693 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.263528 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:48:21.267657 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.267631 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7"] Oct 31 03:48:21.312646 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.312604 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f0902d-8fqf7\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:21.312849 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.312675 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wgh\" (UniqueName: \"kubernetes.io/projected/045b5abd-acde-4a56-8e6e-aecb2f7de383-kube-api-access-d5wgh\") pod \"maas-api-revoke-sub-08f0902d-8fqf7\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:21.312849 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.312714 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f0902d-8fqf7\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:21.313474 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.313448 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f0902d-8fqf7\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:21.315144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.315111 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f0902d-8fqf7\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:21.370362 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.370326 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wgh\" (UniqueName: \"kubernetes.io/projected/045b5abd-acde-4a56-8e6e-aecb2f7de383-kube-api-access-d5wgh\") pod \"maas-api-revoke-sub-08f0902d-8fqf7\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:21.481996 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.481909 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:21.859868 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:21.859835 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7"] Oct 31 03:48:21.860619 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:21.860593 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045b5abd_acde_4a56_8e6e_aecb2f7de383.slice/crio-c933fafd9327f0f9220d488d1312dcb99de4bdbe17b902134b46776c2c4a61e6 WatchSource:0}: Error finding container c933fafd9327f0f9220d488d1312dcb99de4bdbe17b902134b46776c2c4a61e6: Status 404 returned error can't find the container with id c933fafd9327f0f9220d488d1312dcb99de4bdbe17b902134b46776c2c4a61e6 Oct 31 03:48:22.038264 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:22.038232 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" event={"ID":"045b5abd-acde-4a56-8e6e-aecb2f7de383","Type":"ContainerStarted","Data":"c933fafd9327f0f9220d488d1312dcb99de4bdbe17b902134b46776c2c4a61e6"} Oct 31 03:48:23.056890 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.056796 4426 generic.go:356] "Generic (PLEG): container finished" podID="045b5abd-acde-4a56-8e6e-aecb2f7de383" containerID="fdb45fbdb638ff3630f03f4d1f39adcb6581ef97c80276e2472e3f14bf6df765" exitCode=0 Oct 31 03:48:23.057306 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.056883 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" event={"ID":"045b5abd-acde-4a56-8e6e-aecb2f7de383","Type":"ContainerDied","Data":"fdb45fbdb638ff3630f03f4d1f39adcb6581ef97c80276e2472e3f14bf6df765"} Oct 31 03:48:23.259668 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.259639 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:48:23.265207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.265178 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:48:23.502032 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.501984 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx"] Oct 31 03:48:23.507072 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.507041 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:23.560798 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.560760 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx"] Oct 31 03:48:23.639825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.639787 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a24e579-ssssx\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:23.640002 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.639835 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a24e579-ssssx\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:23.640002 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.639983 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnmcb\" (UniqueName: \"kubernetes.io/projected/8147a502-9522-4659-b4d0-aea5b2b9239b-kube-api-access-dnmcb\") pod \"maas-api-revoke-sub-8a24e579-ssssx\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:23.741414 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.741381 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a24e579-ssssx\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:23.741617 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.741430 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a24e579-ssssx\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:23.741617 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.741497 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnmcb\" (UniqueName: \"kubernetes.io/projected/8147a502-9522-4659-b4d0-aea5b2b9239b-kube-api-access-dnmcb\") pod \"maas-api-revoke-sub-8a24e579-ssssx\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:23.742113 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.742085 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a24e579-ssssx\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:23.744162 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.744135 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a24e579-ssssx\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:23.799717 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.799641 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnmcb\" (UniqueName: \"kubernetes.io/projected/8147a502-9522-4659-b4d0-aea5b2b9239b-kube-api-access-dnmcb\") pod \"maas-api-revoke-sub-8a24e579-ssssx\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:23.824694 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:23.824657 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:24.206868 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:24.206830 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx"] Oct 31 03:48:24.207280 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:24.207049 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8147a502_9522_4659_b4d0_aea5b2b9239b.slice/crio-b6c273ea740787251e38e2543a15dc1d9ff4506418211511f707f25b64a09465 WatchSource:0}: Error finding container b6c273ea740787251e38e2543a15dc1d9ff4506418211511f707f25b64a09465: Status 404 returned error can't find the container with id b6c273ea740787251e38e2543a15dc1d9ff4506418211511f707f25b64a09465 Oct 31 03:48:24.232464 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:24.232431 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:24.348577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:24.348486 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-xks-ca\") pod \"045b5abd-acde-4a56-8e6e-aecb2f7de383\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " Oct 31 03:48:24.348737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:24.348650 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045b5abd-acde-4a56-8e6e-aecb2f7de383-kube-api-access-d5wgh\" (UniqueName: \"kubernetes.io/projected/045b5abd-acde-4a56-8e6e-aecb2f7de383-kube-api-access-d5wgh\") pod \"045b5abd-acde-4a56-8e6e-aecb2f7de383\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " Oct 31 03:48:24.348886 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:24.348745 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-openshift-ca\") pod \"045b5abd-acde-4a56-8e6e-aecb2f7de383\" (UID: \"045b5abd-acde-4a56-8e6e-aecb2f7de383\") " Oct 31 03:48:24.349520 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:24.349482 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-openshift-ca" pod "045b5abd-acde-4a56-8e6e-aecb2f7de383" (UID: "045b5abd-acde-4a56-8e6e-aecb2f7de383"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:24.351332 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:24.351302 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045b5abd-acde-4a56-8e6e-aecb2f7de383-kube-api-access-d5wgh" pod "045b5abd-acde-4a56-8e6e-aecb2f7de383" (UID: "045b5abd-acde-4a56-8e6e-aecb2f7de383"). InnerVolumeSpecName "kube-api-access-d5wgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:24.351813 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:24.351789 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-xks-ca" pod "045b5abd-acde-4a56-8e6e-aecb2f7de383" (UID: "045b5abd-acde-4a56-8e6e-aecb2f7de383"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:24.450769 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:24.450724 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:24.450769 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:24.450760 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5wgh\" (UniqueName: \"kubernetes.io/projected/045b5abd-acde-4a56-8e6e-aecb2f7de383-kube-api-access-d5wgh\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:24.450769 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:24.450774 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045b5abd-acde-4a56-8e6e-aecb2f7de383-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:25.074762 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:25.074679 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" Oct 31 03:48:25.074762 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:25.074701 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7" event={"ID":"045b5abd-acde-4a56-8e6e-aecb2f7de383","Type":"ContainerDied","Data":"c933fafd9327f0f9220d488d1312dcb99de4bdbe17b902134b46776c2c4a61e6"} Oct 31 03:48:25.074978 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:25.074783 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c933fafd9327f0f9220d488d1312dcb99de4bdbe17b902134b46776c2c4a61e6" Oct 31 03:48:25.076651 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:25.076622 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" event={"ID":"8147a502-9522-4659-b4d0-aea5b2b9239b","Type":"ContainerStarted","Data":"030327485b876353644874d791a0787abdd7471366c1740323d717e4dd153a7c"} Oct 31 03:48:25.076778 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:25.076678 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" event={"ID":"8147a502-9522-4659-b4d0-aea5b2b9239b","Type":"ContainerStarted","Data":"b6c273ea740787251e38e2543a15dc1d9ff4506418211511f707f25b64a09465"} Oct 31 03:48:25.188493 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:25.188439 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" podStartSLOduration=2.188424266 podStartE2EDuration="2.188424266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:25.185069139 +0000 UTC m=+1202.185694366" watchObservedRunningTime="2026-10-31 03:48:25.188424266 +0000 UTC m=+1202.189049494" Oct 31 03:48:26.086206 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:26.086171 4426 generic.go:356] "Generic (PLEG): container finished" podID="8147a502-9522-4659-b4d0-aea5b2b9239b" containerID="030327485b876353644874d791a0787abdd7471366c1740323d717e4dd153a7c" exitCode=0 Oct 31 03:48:26.086624 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:26.086232 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" event={"ID":"8147a502-9522-4659-b4d0-aea5b2b9239b","Type":"ContainerDied","Data":"030327485b876353644874d791a0787abdd7471366c1740323d717e4dd153a7c"} Oct 31 03:48:27.006237 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.006205 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg"] Oct 31 03:48:27.006797 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.006782 4426 cpu_manager.go:414] "containerMap: removing container" podUID="045b5abd-acde-4a56-8e6e-aecb2f7de383" containerName="revoke-keys" Oct 31 03:48:27.006797 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.006796 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045b5abd-acde-4a56-8e6e-aecb2f7de383" containerName="revoke-keys" Oct 31 03:48:27.006915 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.006880 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="045b5abd-acde-4a56-8e6e-aecb2f7de383" containerName="revoke-keys" Oct 31 03:48:27.010507 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.010489 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:27.071877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.071845 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg"] Oct 31 03:48:27.078812 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.078769 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c621e4b1-qlwwg\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:27.078998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.078922 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frlms\" (UniqueName: \"kubernetes.io/projected/802cbbcb-0123-49f5-80c3-44dfb109cf5d-kube-api-access-frlms\") pod \"maas-api-revoke-sub-c621e4b1-qlwwg\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:27.078998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.078992 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c621e4b1-qlwwg\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:27.184288 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.184254 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frlms\" (UniqueName: \"kubernetes.io/projected/802cbbcb-0123-49f5-80c3-44dfb109cf5d-kube-api-access-frlms\") pod \"maas-api-revoke-sub-c621e4b1-qlwwg\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:27.184765 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.184682 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c621e4b1-qlwwg\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:27.184837 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.184823 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c621e4b1-qlwwg\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:27.185985 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.185938 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c621e4b1-qlwwg\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:27.187966 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.187929 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c621e4b1-qlwwg\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:27.241713 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.241680 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frlms\" (UniqueName: \"kubernetes.io/projected/802cbbcb-0123-49f5-80c3-44dfb109cf5d-kube-api-access-frlms\") pod \"maas-api-revoke-sub-c621e4b1-qlwwg\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:27.271443 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.271407 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:27.323450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.323414 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:27.386846 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.386329 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8147a502-9522-4659-b4d0-aea5b2b9239b-kube-api-access-dnmcb\" (UniqueName: \"kubernetes.io/projected/8147a502-9522-4659-b4d0-aea5b2b9239b-kube-api-access-dnmcb\") pod \"8147a502-9522-4659-b4d0-aea5b2b9239b\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " Oct 31 03:48:27.386846 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.386562 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-xks-ca\") pod \"8147a502-9522-4659-b4d0-aea5b2b9239b\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " Oct 31 03:48:27.387338 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.387315 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-openshift-ca\") pod \"8147a502-9522-4659-b4d0-aea5b2b9239b\" (UID: \"8147a502-9522-4659-b4d0-aea5b2b9239b\") " Oct 31 03:48:27.387711 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.387685 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-openshift-ca" pod "8147a502-9522-4659-b4d0-aea5b2b9239b" (UID: "8147a502-9522-4659-b4d0-aea5b2b9239b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:27.387920 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.387901 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:27.390496 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.390470 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8147a502-9522-4659-b4d0-aea5b2b9239b-kube-api-access-dnmcb" pod "8147a502-9522-4659-b4d0-aea5b2b9239b" (UID: "8147a502-9522-4659-b4d0-aea5b2b9239b"). InnerVolumeSpecName "kube-api-access-dnmcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:27.390615 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.390541 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-xks-ca" pod "8147a502-9522-4659-b4d0-aea5b2b9239b" (UID: "8147a502-9522-4659-b4d0-aea5b2b9239b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:27.486115 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.486078 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv"] Oct 31 03:48:27.486785 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.486764 4426 cpu_manager.go:414] "containerMap: removing container" podUID="8147a502-9522-4659-b4d0-aea5b2b9239b" containerName="revoke-keys" Oct 31 03:48:27.486785 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.486785 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8147a502-9522-4659-b4d0-aea5b2b9239b" containerName="revoke-keys" Oct 31 03:48:27.486951 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.486921 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="8147a502-9522-4659-b4d0-aea5b2b9239b" containerName="revoke-keys" Oct 31 03:48:27.488899 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.488869 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnmcb\" (UniqueName: \"kubernetes.io/projected/8147a502-9522-4659-b4d0-aea5b2b9239b-kube-api-access-dnmcb\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:27.489003 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.488901 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8147a502-9522-4659-b4d0-aea5b2b9239b-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:27.491665 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.491644 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:27.545943 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.545857 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv"] Oct 31 03:48:27.589849 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.589808 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b6ae32c-zjhfv\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:27.590036 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.589942 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmmx\" (UniqueName: \"kubernetes.io/projected/86ea6b9d-9837-485b-aba6-e75eb17df90b-kube-api-access-swmmx\") pod \"maas-api-revoke-sub-9b6ae32c-zjhfv\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:27.590036 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.589992 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b6ae32c-zjhfv\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:27.691080 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.691042 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b6ae32c-zjhfv\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:27.691263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.691144 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmmx\" (UniqueName: \"kubernetes.io/projected/86ea6b9d-9837-485b-aba6-e75eb17df90b-kube-api-access-swmmx\") pod \"maas-api-revoke-sub-9b6ae32c-zjhfv\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:27.691263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.691176 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b6ae32c-zjhfv\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:27.691840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.691815 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b6ae32c-zjhfv\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:27.693454 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.693432 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b6ae32c-zjhfv\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:27.702206 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:27.702179 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802cbbcb_0123_49f5_80c3_44dfb109cf5d.slice/crio-677c632688eb970f8be3d8c34dcf0560a0ecc3346ae96f93586b6b2c9a4ee43e WatchSource:0}: Error finding container 677c632688eb970f8be3d8c34dcf0560a0ecc3346ae96f93586b6b2c9a4ee43e: Status 404 returned error can't find the container with id 677c632688eb970f8be3d8c34dcf0560a0ecc3346ae96f93586b6b2c9a4ee43e Oct 31 03:48:27.702506 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.702483 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg"] Oct 31 03:48:27.748679 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.748646 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmmx\" (UniqueName: \"kubernetes.io/projected/86ea6b9d-9837-485b-aba6-e75eb17df90b-kube-api-access-swmmx\") pod \"maas-api-revoke-sub-9b6ae32c-zjhfv\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:27.810416 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:27.810330 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:28.113971 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:28.113863 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" event={"ID":"802cbbcb-0123-49f5-80c3-44dfb109cf5d","Type":"ContainerStarted","Data":"ef879b07ceac03a00eac604b1ea831c3374fd5f1bc07ed57e63bb4aa95634f52"} Oct 31 03:48:28.113971 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:28.113942 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" event={"ID":"802cbbcb-0123-49f5-80c3-44dfb109cf5d","Type":"ContainerStarted","Data":"677c632688eb970f8be3d8c34dcf0560a0ecc3346ae96f93586b6b2c9a4ee43e"} Oct 31 03:48:28.115541 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:28.115513 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" Oct 31 03:48:28.115541 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:28.115535 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx" event={"ID":"8147a502-9522-4659-b4d0-aea5b2b9239b","Type":"ContainerDied","Data":"b6c273ea740787251e38e2543a15dc1d9ff4506418211511f707f25b64a09465"} Oct 31 03:48:28.115700 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:28.115578 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6c273ea740787251e38e2543a15dc1d9ff4506418211511f707f25b64a09465" Oct 31 03:48:28.187141 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:28.187097 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ea6b9d_9837_485b_aba6_e75eb17df90b.slice/crio-00dfe786139da50f09a63756fbaaa5e9955ea0a1061dff07b02b412565094081 WatchSource:0}: Error finding container 00dfe786139da50f09a63756fbaaa5e9955ea0a1061dff07b02b412565094081: Status 404 returned error can't find the container with id 00dfe786139da50f09a63756fbaaa5e9955ea0a1061dff07b02b412565094081 Oct 31 03:48:28.187454 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:28.187249 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv"] Oct 31 03:48:28.226572 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:28.226515 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" podStartSLOduration=2.226502113 podStartE2EDuration="2.226502113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:28.222434242 +0000 UTC m=+1205.223059469" watchObservedRunningTime="2026-10-31 03:48:28.226502113 +0000 UTC m=+1205.227127372" Oct 31 03:48:29.128503 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:29.128410 4426 generic.go:356] "Generic (PLEG): container finished" podID="802cbbcb-0123-49f5-80c3-44dfb109cf5d" containerID="ef879b07ceac03a00eac604b1ea831c3374fd5f1bc07ed57e63bb4aa95634f52" exitCode=0 Oct 31 03:48:29.131584 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:29.131557 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" event={"ID":"86ea6b9d-9837-485b-aba6-e75eb17df90b","Type":"ContainerStarted","Data":"cd2267adb2fb70f391c7c1d698d73b428d118f6e4d8a8e84525c47d7cbb2fab9"} Oct 31 03:48:29.131717 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:29.131646 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" event={"ID":"86ea6b9d-9837-485b-aba6-e75eb17df90b","Type":"ContainerStarted","Data":"00dfe786139da50f09a63756fbaaa5e9955ea0a1061dff07b02b412565094081"} Oct 31 03:48:29.131717 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:29.131666 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" event={"ID":"802cbbcb-0123-49f5-80c3-44dfb109cf5d","Type":"ContainerDied","Data":"ef879b07ceac03a00eac604b1ea831c3374fd5f1bc07ed57e63bb4aa95634f52"} Oct 31 03:48:29.237443 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:29.237387 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" podStartSLOduration=2.237371641 podStartE2EDuration="2.237371641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:29.236267523 +0000 UTC m=+1206.236892752" watchObservedRunningTime="2026-10-31 03:48:29.237371641 +0000 UTC m=+1206.237996869" Oct 31 03:48:30.141037 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.141000 4426 generic.go:356] "Generic (PLEG): container finished" podID="86ea6b9d-9837-485b-aba6-e75eb17df90b" containerID="cd2267adb2fb70f391c7c1d698d73b428d118f6e4d8a8e84525c47d7cbb2fab9" exitCode=0 Oct 31 03:48:30.141256 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.141083 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" event={"ID":"86ea6b9d-9837-485b-aba6-e75eb17df90b","Type":"ContainerDied","Data":"cd2267adb2fb70f391c7c1d698d73b428d118f6e4d8a8e84525c47d7cbb2fab9"} Oct 31 03:48:30.192033 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.191995 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wr6bj-n6rpw"] Oct 31 03:48:30.196228 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.196208 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.252252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.251884 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wr6bj-n6rpw"] Oct 31 03:48:30.252673 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.252348 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 03:48:30.298815 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.298790 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bzhcg\"" Oct 31 03:48:30.321052 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.321025 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:30.334236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.331235 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndc8\" (UniqueName: \"kubernetes.io/projected/507c0207-cbfd-48a6-b463-5e7b615589b6-kube-api-access-zndc8\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.334236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.331292 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/507c0207-cbfd-48a6-b463-5e7b615589b6-archives-path\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.334236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.331320 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-service-ca-bundle\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.334236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.331445 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-trusted-ca-bundle\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.346105 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.346064 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 03:48:30.432656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.432557 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-xks-ca\") pod \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " Oct 31 03:48:30.432836 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.432689 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802cbbcb-0123-49f5-80c3-44dfb109cf5d-kube-api-access-frlms\" (UniqueName: \"kubernetes.io/projected/802cbbcb-0123-49f5-80c3-44dfb109cf5d-kube-api-access-frlms\") pod \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " Oct 31 03:48:30.432836 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.432823 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-openshift-ca\") pod \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\" (UID: \"802cbbcb-0123-49f5-80c3-44dfb109cf5d\") " Oct 31 03:48:30.433089 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.433060 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndc8\" (UniqueName: \"kubernetes.io/projected/507c0207-cbfd-48a6-b463-5e7b615589b6-kube-api-access-zndc8\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.433236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.433111 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/507c0207-cbfd-48a6-b463-5e7b615589b6-archives-path\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.433236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.433156 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-service-ca-bundle\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.433362 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.433245 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-openshift-ca" pod "802cbbcb-0123-49f5-80c3-44dfb109cf5d" (UID: "802cbbcb-0123-49f5-80c3-44dfb109cf5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:30.433362 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.433266 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-trusted-ca-bundle\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.433477 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.433385 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:30.433625 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.433597 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/507c0207-cbfd-48a6-b463-5e7b615589b6-archives-path\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.434246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.434207 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-service-ca-bundle\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.434246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.434217 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-trusted-ca-bundle\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.435256 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.435227 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-xks-ca" pod "802cbbcb-0123-49f5-80c3-44dfb109cf5d" (UID: "802cbbcb-0123-49f5-80c3-44dfb109cf5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:30.435366 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.435258 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802cbbcb-0123-49f5-80c3-44dfb109cf5d-kube-api-access-frlms" pod "802cbbcb-0123-49f5-80c3-44dfb109cf5d" (UID: "802cbbcb-0123-49f5-80c3-44dfb109cf5d"). InnerVolumeSpecName "kube-api-access-frlms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:30.491910 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.491874 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndc8\" (UniqueName: \"kubernetes.io/projected/507c0207-cbfd-48a6-b463-5e7b615589b6-kube-api-access-zndc8\") pod \"periodic-gathering-wr6bj-n6rpw\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.512072 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.512037 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:48:30.534503 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.534470 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802cbbcb-0123-49f5-80c3-44dfb109cf5d-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:30.534644 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.534506 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frlms\" (UniqueName: \"kubernetes.io/projected/802cbbcb-0123-49f5-80c3-44dfb109cf5d-kube-api-access-frlms\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:30.897433 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:30.897402 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wr6bj-n6rpw"] Oct 31 03:48:30.898050 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:30.898026 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507c0207_cbfd_48a6_b463_5e7b615589b6.slice/crio-ad08440b938a6290d6db2c67745a66d49c4666638d3f9c70ac1ba28fa6098030 WatchSource:0}: Error finding container ad08440b938a6290d6db2c67745a66d49c4666638d3f9c70ac1ba28fa6098030: Status 404 returned error can't find the container with id ad08440b938a6290d6db2c67745a66d49c4666638d3f9c70ac1ba28fa6098030 Oct 31 03:48:31.155219 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.155109 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" Oct 31 03:48:31.155219 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.155110 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg" event={"ID":"802cbbcb-0123-49f5-80c3-44dfb109cf5d","Type":"ContainerDied","Data":"677c632688eb970f8be3d8c34dcf0560a0ecc3346ae96f93586b6b2c9a4ee43e"} Oct 31 03:48:31.155442 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.155240 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="677c632688eb970f8be3d8c34dcf0560a0ecc3346ae96f93586b6b2c9a4ee43e" Oct 31 03:48:31.157739 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.157629 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" event={"ID":"507c0207-cbfd-48a6-b463-5e7b615589b6","Type":"ContainerStarted","Data":"dd0614a4b3a1d5647c34ecb0e33dc8ff04bf57658820b537c886aa3f3d61161d"} Oct 31 03:48:31.157739 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.157678 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" event={"ID":"507c0207-cbfd-48a6-b463-5e7b615589b6","Type":"ContainerStarted","Data":"ad08440b938a6290d6db2c67745a66d49c4666638d3f9c70ac1ba28fa6098030"} Oct 31 03:48:31.271724 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.271660 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" podStartSLOduration=1.271641973 podStartE2EDuration="1.271641973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:31.265776267 +0000 UTC m=+1208.266401495" watchObservedRunningTime="2026-10-31 03:48:31.271641973 +0000 UTC m=+1208.272267202" Oct 31 03:48:31.308175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.308144 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:31.448054 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.447962 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-openshift-ca\") pod \"86ea6b9d-9837-485b-aba6-e75eb17df90b\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " Oct 31 03:48:31.448271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.448074 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-xks-ca\") pod \"86ea6b9d-9837-485b-aba6-e75eb17df90b\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " Oct 31 03:48:31.448271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.448194 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ea6b9d-9837-485b-aba6-e75eb17df90b-kube-api-access-swmmx\" (UniqueName: \"kubernetes.io/projected/86ea6b9d-9837-485b-aba6-e75eb17df90b-kube-api-access-swmmx\") pod \"86ea6b9d-9837-485b-aba6-e75eb17df90b\" (UID: \"86ea6b9d-9837-485b-aba6-e75eb17df90b\") " Oct 31 03:48:31.448416 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.448352 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-openshift-ca" pod "86ea6b9d-9837-485b-aba6-e75eb17df90b" (UID: "86ea6b9d-9837-485b-aba6-e75eb17df90b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:31.448685 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.448662 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:31.450978 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.450949 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-xks-ca" pod "86ea6b9d-9837-485b-aba6-e75eb17df90b" (UID: "86ea6b9d-9837-485b-aba6-e75eb17df90b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:31.456283 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.456256 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ea6b9d-9837-485b-aba6-e75eb17df90b-kube-api-access-swmmx" pod "86ea6b9d-9837-485b-aba6-e75eb17df90b" (UID: "86ea6b9d-9837-485b-aba6-e75eb17df90b"). InnerVolumeSpecName "kube-api-access-swmmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:31.550415 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.550378 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ea6b9d-9837-485b-aba6-e75eb17df90b-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:31.550415 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:31.550409 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swmmx\" (UniqueName: \"kubernetes.io/projected/86ea6b9d-9837-485b-aba6-e75eb17df90b-kube-api-access-swmmx\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:32.166678 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:32.166645 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" Oct 31 03:48:32.166858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:32.166680 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv" event={"ID":"86ea6b9d-9837-485b-aba6-e75eb17df90b","Type":"ContainerDied","Data":"00dfe786139da50f09a63756fbaaa5e9955ea0a1061dff07b02b412565094081"} Oct 31 03:48:32.166858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:32.166739 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00dfe786139da50f09a63756fbaaa5e9955ea0a1061dff07b02b412565094081" Oct 31 03:48:32.236964 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:32.236920 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:48:32.856081 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:48:32.856036 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:48:37.771633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.771553 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z"] Oct 31 03:48:37.772204 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.772184 4426 cpu_manager.go:414] "containerMap: removing container" podUID="86ea6b9d-9837-485b-aba6-e75eb17df90b" containerName="revoke-keys" Oct 31 03:48:37.772265 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.772204 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ea6b9d-9837-485b-aba6-e75eb17df90b" containerName="revoke-keys" Oct 31 03:48:37.772265 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.772231 4426 cpu_manager.go:414] "containerMap: removing container" podUID="802cbbcb-0123-49f5-80c3-44dfb109cf5d" containerName="revoke-keys" Oct 31 03:48:37.772265 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.772236 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802cbbcb-0123-49f5-80c3-44dfb109cf5d" containerName="revoke-keys" Oct 31 03:48:37.772382 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.772332 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ea6b9d-9837-485b-aba6-e75eb17df90b" containerName="revoke-keys" Oct 31 03:48:37.772382 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.772345 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="802cbbcb-0123-49f5-80c3-44dfb109cf5d" containerName="revoke-keys" Oct 31 03:48:37.776195 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.776172 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:37.825796 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.825762 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:48:37.825796 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.825795 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:48:37.826041 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.825851 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:48:37.834036 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.833997 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8fae9e4-qfr8z\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:37.834263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.834231 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8fae9e4-qfr8z\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:37.834413 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.834389 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzg2\" (UniqueName: \"kubernetes.io/projected/334d46da-9dcf-45c1-9709-14200152e8b4-kube-api-access-hhzg2\") pod \"maas-api-revoke-sub-f8fae9e4-qfr8z\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:37.872465 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.872428 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:48:37.879323 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.879289 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z"] Oct 31 03:48:37.935453 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.935411 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8fae9e4-qfr8z\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:37.935665 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.935509 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzg2\" (UniqueName: \"kubernetes.io/projected/334d46da-9dcf-45c1-9709-14200152e8b4-kube-api-access-hhzg2\") pod \"maas-api-revoke-sub-f8fae9e4-qfr8z\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:37.935665 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.935597 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8fae9e4-qfr8z\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:37.936303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.936275 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8fae9e4-qfr8z\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:37.937998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.937972 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8fae9e4-qfr8z\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:37.993270 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:37.993231 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzg2\" (UniqueName: \"kubernetes.io/projected/334d46da-9dcf-45c1-9709-14200152e8b4-kube-api-access-hhzg2\") pod \"maas-api-revoke-sub-f8fae9e4-qfr8z\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:38.090555 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:38.090473 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:38.473013 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:38.472983 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334d46da_9dcf_45c1_9709_14200152e8b4.slice/crio-947e5ad6064fbd948c7143a284d086088ae4bad1cf10282bed23580f0b742859 WatchSource:0}: Error finding container 947e5ad6064fbd948c7143a284d086088ae4bad1cf10282bed23580f0b742859: Status 404 returned error can't find the container with id 947e5ad6064fbd948c7143a284d086088ae4bad1cf10282bed23580f0b742859 Oct 31 03:48:38.473396 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:38.473371 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z"] Oct 31 03:48:39.235411 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.235376 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" event={"ID":"334d46da-9dcf-45c1-9709-14200152e8b4","Type":"ContainerStarted","Data":"f2c41ed583c8e96bff6b3397fbbdd8696dca7a977769872fd5f3635114785867"} Oct 31 03:48:39.235824 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.235463 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" event={"ID":"334d46da-9dcf-45c1-9709-14200152e8b4","Type":"ContainerStarted","Data":"947e5ad6064fbd948c7143a284d086088ae4bad1cf10282bed23580f0b742859"} Oct 31 03:48:39.345810 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.345742 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" podStartSLOduration=2.345728805 podStartE2EDuration="2.345728805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:39.344040486 +0000 UTC m=+1216.344665716" watchObservedRunningTime="2026-10-31 03:48:39.345728805 +0000 UTC m=+1216.346354036" Oct 31 03:48:39.689371 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.689337 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx"] Oct 31 03:48:39.693654 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.693633 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:39.753756 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.753721 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx"] Oct 31 03:48:39.758807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.758775 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298bde85-443f-4237-8e71-7737b7630618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9650182-6klnx\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:39.759009 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.758983 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jzn\" (UniqueName: \"kubernetes.io/projected/298bde85-443f-4237-8e71-7737b7630618-kube-api-access-28jzn\") pod \"maas-api-revoke-sub-d9650182-6klnx\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:39.759111 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.759052 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298bde85-443f-4237-8e71-7737b7630618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9650182-6klnx\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:39.860658 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.860621 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298bde85-443f-4237-8e71-7737b7630618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9650182-6klnx\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:39.860847 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.860677 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jzn\" (UniqueName: \"kubernetes.io/projected/298bde85-443f-4237-8e71-7737b7630618-kube-api-access-28jzn\") pod \"maas-api-revoke-sub-d9650182-6klnx\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:39.860847 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.860726 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298bde85-443f-4237-8e71-7737b7630618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9650182-6klnx\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:39.861421 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.861395 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298bde85-443f-4237-8e71-7737b7630618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9650182-6klnx\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:39.863142 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.863105 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298bde85-443f-4237-8e71-7737b7630618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9650182-6klnx\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:39.918541 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:39.918509 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jzn\" (UniqueName: \"kubernetes.io/projected/298bde85-443f-4237-8e71-7737b7630618-kube-api-access-28jzn\") pod \"maas-api-revoke-sub-d9650182-6klnx\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:40.007602 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:40.007511 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:40.245194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:40.245162 4426 generic.go:356] "Generic (PLEG): container finished" podID="334d46da-9dcf-45c1-9709-14200152e8b4" containerID="f2c41ed583c8e96bff6b3397fbbdd8696dca7a977769872fd5f3635114785867" exitCode=0 Oct 31 03:48:40.245637 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:40.245251 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" event={"ID":"334d46da-9dcf-45c1-9709-14200152e8b4","Type":"ContainerDied","Data":"f2c41ed583c8e96bff6b3397fbbdd8696dca7a977769872fd5f3635114785867"} Oct 31 03:48:40.391917 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:40.391886 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298bde85_443f_4237_8e71_7737b7630618.slice/crio-a342877ef7ec63bfc32d255151666b199bbf4a1f318e5a4c552b2b522b263f8c WatchSource:0}: Error finding container a342877ef7ec63bfc32d255151666b199bbf4a1f318e5a4c552b2b522b263f8c: Status 404 returned error can't find the container with id a342877ef7ec63bfc32d255151666b199bbf4a1f318e5a4c552b2b522b263f8c Oct 31 03:48:40.405205 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:40.405177 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx"] Oct 31 03:48:41.254199 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.254160 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" event={"ID":"298bde85-443f-4237-8e71-7737b7630618","Type":"ContainerStarted","Data":"dc19b07d4f302b577227a69c8056ac7bdb1305b00ca885a7db4347691e9b88dd"} Oct 31 03:48:41.254655 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.254222 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" event={"ID":"298bde85-443f-4237-8e71-7737b7630618","Type":"ContainerStarted","Data":"a342877ef7ec63bfc32d255151666b199bbf4a1f318e5a4c552b2b522b263f8c"} Oct 31 03:48:41.367658 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.367594 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" podStartSLOduration=2.367573033 podStartE2EDuration="2.367573033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:41.361940341 +0000 UTC m=+1218.362565571" watchObservedRunningTime="2026-10-31 03:48:41.367573033 +0000 UTC m=+1218.368198262" Oct 31 03:48:41.415434 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.415400 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:41.482006 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.481971 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334d46da-9dcf-45c1-9709-14200152e8b4-kube-api-access-hhzg2\" (UniqueName: \"kubernetes.io/projected/334d46da-9dcf-45c1-9709-14200152e8b4-kube-api-access-hhzg2\") pod \"334d46da-9dcf-45c1-9709-14200152e8b4\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " Oct 31 03:48:41.482235 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.482032 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-xks-ca\") pod \"334d46da-9dcf-45c1-9709-14200152e8b4\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " Oct 31 03:48:41.482235 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.482068 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-openshift-ca\") pod \"334d46da-9dcf-45c1-9709-14200152e8b4\" (UID: \"334d46da-9dcf-45c1-9709-14200152e8b4\") " Oct 31 03:48:41.482498 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.482461 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-openshift-ca" pod "334d46da-9dcf-45c1-9709-14200152e8b4" (UID: "334d46da-9dcf-45c1-9709-14200152e8b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:41.484354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.484328 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-xks-ca" pod "334d46da-9dcf-45c1-9709-14200152e8b4" (UID: "334d46da-9dcf-45c1-9709-14200152e8b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:41.484461 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.484329 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334d46da-9dcf-45c1-9709-14200152e8b4-kube-api-access-hhzg2" pod "334d46da-9dcf-45c1-9709-14200152e8b4" (UID: "334d46da-9dcf-45c1-9709-14200152e8b4"). InnerVolumeSpecName "kube-api-access-hhzg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:41.584178 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.584056 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzg2\" (UniqueName: \"kubernetes.io/projected/334d46da-9dcf-45c1-9709-14200152e8b4-kube-api-access-hhzg2\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:41.584178 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.584089 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:41.584178 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:41.584099 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334d46da-9dcf-45c1-9709-14200152e8b4-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:42.240972 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.240927 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj"] Oct 31 03:48:42.241640 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.241619 4426 cpu_manager.go:414] "containerMap: removing container" podUID="334d46da-9dcf-45c1-9709-14200152e8b4" containerName="revoke-keys" Oct 31 03:48:42.241640 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.241641 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334d46da-9dcf-45c1-9709-14200152e8b4" containerName="revoke-keys" Oct 31 03:48:42.241835 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.241822 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="334d46da-9dcf-45c1-9709-14200152e8b4" containerName="revoke-keys" Oct 31 03:48:42.245378 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.245354 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:42.264661 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.264624 4426 generic.go:356] "Generic (PLEG): container finished" podID="298bde85-443f-4237-8e71-7737b7630618" containerID="dc19b07d4f302b577227a69c8056ac7bdb1305b00ca885a7db4347691e9b88dd" exitCode=0 Oct 31 03:48:42.265073 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.264708 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" event={"ID":"298bde85-443f-4237-8e71-7737b7630618","Type":"ContainerDied","Data":"dc19b07d4f302b577227a69c8056ac7bdb1305b00ca885a7db4347691e9b88dd"} Oct 31 03:48:42.266114 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.266090 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" event={"ID":"334d46da-9dcf-45c1-9709-14200152e8b4","Type":"ContainerDied","Data":"947e5ad6064fbd948c7143a284d086088ae4bad1cf10282bed23580f0b742859"} Oct 31 03:48:42.266232 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.266111 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z" Oct 31 03:48:42.266232 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.266149 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="947e5ad6064fbd948c7143a284d086088ae4bad1cf10282bed23580f0b742859" Oct 31 03:48:42.293114 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.293085 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53c9f57-j5gfj\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:42.293274 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.293231 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53c9f57-j5gfj\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:42.293333 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.293307 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rw2\" (UniqueName: \"kubernetes.io/projected/a3ea1f60-1ffe-43e3-899b-6925af1d0608-kube-api-access-r7rw2\") pod \"maas-api-revoke-sub-a53c9f57-j5gfj\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:42.301962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.301922 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj"] Oct 31 03:48:42.394892 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.394853 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53c9f57-j5gfj\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:42.395151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.394967 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53c9f57-j5gfj\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:42.395151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.395001 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rw2\" (UniqueName: \"kubernetes.io/projected/a3ea1f60-1ffe-43e3-899b-6925af1d0608-kube-api-access-r7rw2\") pod \"maas-api-revoke-sub-a53c9f57-j5gfj\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:42.395826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.395783 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53c9f57-j5gfj\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:42.397323 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.397298 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53c9f57-j5gfj\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:42.452856 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.452820 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rw2\" (UniqueName: \"kubernetes.io/projected/a3ea1f60-1ffe-43e3-899b-6925af1d0608-kube-api-access-r7rw2\") pod \"maas-api-revoke-sub-a53c9f57-j5gfj\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:42.561629 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.561536 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:42.951446 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:42.951401 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj"] Oct 31 03:48:42.957039 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:42.957003 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ea1f60_1ffe_43e3_899b_6925af1d0608.slice/crio-f279cae1c65194b44cd30f2747c5e830396a13ebcb6f40a9466714ebbbb014a3 WatchSource:0}: Error finding container f279cae1c65194b44cd30f2747c5e830396a13ebcb6f40a9466714ebbbb014a3: Status 404 returned error can't find the container with id f279cae1c65194b44cd30f2747c5e830396a13ebcb6f40a9466714ebbbb014a3 Oct 31 03:48:43.275197 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.275161 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" event={"ID":"a3ea1f60-1ffe-43e3-899b-6925af1d0608","Type":"ContainerStarted","Data":"1dcec9433e6d743816dec677c899f18525b02362abcd6c1adce5b4d404988f75"} Oct 31 03:48:43.275620 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.275227 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" event={"ID":"a3ea1f60-1ffe-43e3-899b-6925af1d0608","Type":"ContainerStarted","Data":"f279cae1c65194b44cd30f2747c5e830396a13ebcb6f40a9466714ebbbb014a3"} Oct 31 03:48:43.388579 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.388521 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" podStartSLOduration=1.3885074419999999 podStartE2EDuration="1.388507442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:43.382733961 +0000 UTC m=+1220.383359190" watchObservedRunningTime="2026-10-31 03:48:43.388507442 +0000 UTC m=+1220.389132670" Oct 31 03:48:43.442967 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.442932 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:43.508333 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.508239 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298bde85-443f-4237-8e71-7737b7630618-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298bde85-443f-4237-8e71-7737b7630618-maas-api-openshift-ca\") pod \"298bde85-443f-4237-8e71-7737b7630618\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " Oct 31 03:48:43.508511 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.508360 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298bde85-443f-4237-8e71-7737b7630618-kube-api-access-28jzn\" (UniqueName: \"kubernetes.io/projected/298bde85-443f-4237-8e71-7737b7630618-kube-api-access-28jzn\") pod \"298bde85-443f-4237-8e71-7737b7630618\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " Oct 31 03:48:43.508582 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.508533 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/298bde85-443f-4237-8e71-7737b7630618-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298bde85-443f-4237-8e71-7737b7630618-maas-api-xks-ca\") pod \"298bde85-443f-4237-8e71-7737b7630618\" (UID: \"298bde85-443f-4237-8e71-7737b7630618\") " Oct 31 03:48:43.508674 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.508647 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298bde85-443f-4237-8e71-7737b7630618-maas-api-openshift-ca" pod "298bde85-443f-4237-8e71-7737b7630618" (UID: "298bde85-443f-4237-8e71-7737b7630618"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:43.508980 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.508961 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298bde85-443f-4237-8e71-7737b7630618-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:43.510554 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.510527 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298bde85-443f-4237-8e71-7737b7630618-kube-api-access-28jzn" pod "298bde85-443f-4237-8e71-7737b7630618" (UID: "298bde85-443f-4237-8e71-7737b7630618"). InnerVolumeSpecName "kube-api-access-28jzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:43.510651 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.510572 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/298bde85-443f-4237-8e71-7737b7630618-maas-api-xks-ca" pod "298bde85-443f-4237-8e71-7737b7630618" (UID: "298bde85-443f-4237-8e71-7737b7630618"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:43.609698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.609664 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298bde85-443f-4237-8e71-7737b7630618-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:43.609698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:43.609697 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28jzn\" (UniqueName: \"kubernetes.io/projected/298bde85-443f-4237-8e71-7737b7630618-kube-api-access-28jzn\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:44.284203 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:44.284174 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" Oct 31 03:48:44.284687 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:44.284160 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx" event={"ID":"298bde85-443f-4237-8e71-7737b7630618","Type":"ContainerDied","Data":"a342877ef7ec63bfc32d255151666b199bbf4a1f318e5a4c552b2b522b263f8c"} Oct 31 03:48:44.284687 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:44.284298 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a342877ef7ec63bfc32d255151666b199bbf4a1f318e5a4c552b2b522b263f8c" Oct 31 03:48:44.286066 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:44.286038 4426 generic.go:356] "Generic (PLEG): container finished" podID="a3ea1f60-1ffe-43e3-899b-6925af1d0608" containerID="1dcec9433e6d743816dec677c899f18525b02362abcd6c1adce5b4d404988f75" exitCode=0 Oct 31 03:48:44.286217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:44.286157 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" event={"ID":"a3ea1f60-1ffe-43e3-899b-6925af1d0608","Type":"ContainerDied","Data":"1dcec9433e6d743816dec677c899f18525b02362abcd6c1adce5b4d404988f75"} Oct 31 03:48:45.464176 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:45.464135 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:45.533840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:45.533795 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-xks-ca\") pod \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " Oct 31 03:48:45.534019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:45.533989 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-openshift-ca\") pod \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " Oct 31 03:48:45.534075 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:45.534022 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ea1f60-1ffe-43e3-899b-6925af1d0608-kube-api-access-r7rw2\" (UniqueName: \"kubernetes.io/projected/a3ea1f60-1ffe-43e3-899b-6925af1d0608-kube-api-access-r7rw2\") pod \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\" (UID: \"a3ea1f60-1ffe-43e3-899b-6925af1d0608\") " Oct 31 03:48:45.534397 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:45.534355 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-openshift-ca" pod "a3ea1f60-1ffe-43e3-899b-6925af1d0608" (UID: "a3ea1f60-1ffe-43e3-899b-6925af1d0608"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:45.534527 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:45.534508 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:45.535967 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:45.535944 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-xks-ca" pod "a3ea1f60-1ffe-43e3-899b-6925af1d0608" (UID: "a3ea1f60-1ffe-43e3-899b-6925af1d0608"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:45.536074 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:45.536058 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ea1f60-1ffe-43e3-899b-6925af1d0608-kube-api-access-r7rw2" pod "a3ea1f60-1ffe-43e3-899b-6925af1d0608" (UID: "a3ea1f60-1ffe-43e3-899b-6925af1d0608"). InnerVolumeSpecName "kube-api-access-r7rw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:45.635801 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:45.635710 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rw2\" (UniqueName: \"kubernetes.io/projected/a3ea1f60-1ffe-43e3-899b-6925af1d0608-kube-api-access-r7rw2\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:45.635801 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:45.635743 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ea1f60-1ffe-43e3-899b-6925af1d0608-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:46.304175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:46.304141 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" Oct 31 03:48:46.304175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:46.304119 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj" event={"ID":"a3ea1f60-1ffe-43e3-899b-6925af1d0608","Type":"ContainerDied","Data":"f279cae1c65194b44cd30f2747c5e830396a13ebcb6f40a9466714ebbbb014a3"} Oct 31 03:48:46.304388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:46.304207 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f279cae1c65194b44cd30f2747c5e830396a13ebcb6f40a9466714ebbbb014a3" Oct 31 03:48:50.790656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.790601 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl"] Oct 31 03:48:50.791250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.791234 4426 cpu_manager.go:414] "containerMap: removing container" podUID="a3ea1f60-1ffe-43e3-899b-6925af1d0608" containerName="revoke-keys" Oct 31 03:48:50.791250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.791250 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ea1f60-1ffe-43e3-899b-6925af1d0608" containerName="revoke-keys" Oct 31 03:48:50.791354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.791263 4426 cpu_manager.go:414] "containerMap: removing container" podUID="298bde85-443f-4237-8e71-7737b7630618" containerName="revoke-keys" Oct 31 03:48:50.791354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.791269 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298bde85-443f-4237-8e71-7737b7630618" containerName="revoke-keys" Oct 31 03:48:50.791354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.791346 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ea1f60-1ffe-43e3-899b-6925af1d0608" containerName="revoke-keys" Oct 31 03:48:50.791449 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.791358 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="298bde85-443f-4237-8e71-7737b7630618" containerName="revoke-keys" Oct 31 03:48:50.795342 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.795323 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:50.844527 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.844495 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ae8e27-openshift-default-dockercfg-sgjl6\"" Oct 31 03:48:50.897452 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.897414 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl"] Oct 31 03:48:50.900715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.900684 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:50.900878 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.900728 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-certs\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:50.900878 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.900752 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-token\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:50.900878 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.900829 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istiod-ca-cert\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:50.901057 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.900888 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmg7b\" (UniqueName: \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-kube-api-access-rmg7b\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:50.901057 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.900915 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-credential-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:50.901057 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.900978 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-envoy\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:50.901057 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.901009 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-data\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:50.901057 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:50.901028 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-podinfo\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.002516 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.002467 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.002696 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.002535 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-certs\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.002696 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.002570 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-token\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.002696 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.002587 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istiod-ca-cert\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.002696 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.002609 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmg7b\" (UniqueName: \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-kube-api-access-rmg7b\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.002696 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.002649 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-credential-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.002696 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.002683 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-envoy\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.002900 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.002714 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-data\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.002900 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.002732 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-podinfo\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.003528 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.003501 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-credential-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.003744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.003720 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.003969 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.003949 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istiod-ca-cert\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.004294 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.004262 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-certs\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.004691 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.004667 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-data\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.007234 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.005831 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-podinfo\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.012300 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.007737 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-envoy\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.062227 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.062151 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-token\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.062357 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.062309 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmg7b\" (UniqueName: \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-kube-api-access-rmg7b\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.109408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.109374 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:51.139091 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.139055 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v"] Oct 31 03:48:51.139537 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.139490 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" containerID="cri-o://6e020e444d920e041f3a7d6b6f5b9762633b95c7a6058311dd47526e6355ec47" gracePeriod=30 Oct 31 03:48:51.575795 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.575755 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl"] Oct 31 03:48:51.577333 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.577289 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:48:51.577457 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.577378 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:48:51.577457 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.577425 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:48:51.889217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.886647 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp"] Oct 31 03:48:51.897723 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.891079 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:51.993758 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:51.993628 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp"] Oct 31 03:48:52.034351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.033940 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-istio-token\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.034351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.033982 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-credential-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.034351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.034002 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8qx2\" (UniqueName: \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-kube-api-access-s8qx2\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.034351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.034032 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89149831-510f-4dbb-8633-63176576a247-istiod-ca-cert\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.034351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.034061 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-envoy\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.034351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.034085 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-data\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.034351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.034108 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.034351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.034178 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89149831-510f-4dbb-8633-63176576a247-istio-podinfo\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.034351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.034199 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-certs\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135072 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-data\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135159 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135247 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89149831-510f-4dbb-8633-63176576a247-istio-podinfo\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135278 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-certs\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135488 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135408 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-istio-token\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135488 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135437 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-credential-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135488 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135465 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8qx2\" (UniqueName: \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-kube-api-access-s8qx2\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135517 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89149831-510f-4dbb-8633-63176576a247-istiod-ca-cert\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135559 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-data\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135573 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135566 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-envoy\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.135881 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.135845 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-credential-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.136385 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.136358 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-certs\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.136490 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.136424 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89149831-510f-4dbb-8633-63176576a247-istiod-ca-cert\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.138155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.138092 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-envoy\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.138818 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.138798 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89149831-510f-4dbb-8633-63176576a247-istio-podinfo\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.193446 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.193410 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8qx2\" (UniqueName: \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-kube-api-access-s8qx2\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.193660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.193415 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-istio-token\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.212261 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.212210 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:52.386902 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.386515 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" event={"ID":"b790d330-2c37-4ce4-8cf3-57afa5e12d63","Type":"ContainerStarted","Data":"70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099"} Oct 31 03:48:52.386902 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.386581 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" event={"ID":"b790d330-2c37-4ce4-8cf3-57afa5e12d63","Type":"ContainerStarted","Data":"306bb1471117ef7994b5d9d990cbd85da373b022c7ae43ab9be1cd0697394ed2"} Oct 31 03:48:52.552617 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.552440 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" podStartSLOduration=2.552419495 podStartE2EDuration="2.552419495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:52.545772892 +0000 UTC m=+1229.546398123" watchObservedRunningTime="2026-10-31 03:48:52.552419495 +0000 UTC m=+1229.553044726" Oct 31 03:48:52.683197 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:48:52.683160 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89149831_510f_4dbb_8633_63176576a247.slice/crio-7528ae6b42232d3b22e68bf1cc57828a1d8893e92beb93d2a4ea139ecd482071 WatchSource:0}: Error finding container 7528ae6b42232d3b22e68bf1cc57828a1d8893e92beb93d2a4ea139ecd482071: Status 404 returned error can't find the container with id 7528ae6b42232d3b22e68bf1cc57828a1d8893e92beb93d2a4ea139ecd482071 Oct 31 03:48:52.683614 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.683590 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp"] Oct 31 03:48:52.685948 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.685916 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:48:52.686029 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.685980 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:48:52.686029 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:52.686010 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:48:53.109903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:53.109870 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:53.111745 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:53.111711 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:48:53.111897 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:53.111768 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:48:53.397037 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:53.397002 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" event={"ID":"89149831-510f-4dbb-8633-63176576a247","Type":"ContainerStarted","Data":"ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816"} Oct 31 03:48:53.397254 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:53.397060 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" event={"ID":"89149831-510f-4dbb-8633-63176576a247","Type":"ContainerStarted","Data":"7528ae6b42232d3b22e68bf1cc57828a1d8893e92beb93d2a4ea139ecd482071"} Oct 31 03:48:53.564698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:53.564629 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" podStartSLOduration=2.56460547 podStartE2EDuration="2.56460547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:53.544510118 +0000 UTC m=+1230.545135361" watchObservedRunningTime="2026-10-31 03:48:53.56460547 +0000 UTC m=+1230.565230700" Oct 31 03:48:54.110753 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:54.110720 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:48:54.111202 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:54.110775 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:48:54.213961 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:54.213918 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:48:54.244149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:54.215116 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:48:54.244149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:54.215195 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:48:55.110107 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:55.110059 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:48:55.110340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:55.110113 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:48:55.213153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:55.212719 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:48:55.213153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:55.212788 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:48:56.109969 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.109937 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:48:56.110185 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.109991 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:48:56.213565 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.213529 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:48:56.213946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.213593 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:48:56.484959 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.484930 4426 generic.go:356] "Generic (PLEG): container finished" podID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerID="6e020e444d920e041f3a7d6b6f5b9762633b95c7a6058311dd47526e6355ec47" exitCode=0 Oct 31 03:48:56.485411 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.485389 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" event={"ID":"7ec938ed-e85d-42fe-a99c-301a2b361fef","Type":"ContainerDied","Data":"6e020e444d920e041f3a7d6b6f5b9762633b95c7a6058311dd47526e6355ec47"} Oct 31 03:48:56.508219 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.508188 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:48:56.606750 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.606718 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-socket\") pod \"7ec938ed-e85d-42fe-a99c-301a2b361fef\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " Oct 31 03:48:56.606905 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.606791 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-kube-api-access-qbvb8\" (UniqueName: \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-kube-api-access-qbvb8\") pod \"7ec938ed-e85d-42fe-a99c-301a2b361fef\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " Oct 31 03:48:56.606905 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.606814 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-envoy\") pod \"7ec938ed-e85d-42fe-a99c-301a2b361fef\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " Oct 31 03:48:56.607000 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.606975 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-socket" pod "7ec938ed-e85d-42fe-a99c-301a2b361fef" (UID: "7ec938ed-e85d-42fe-a99c-301a2b361fef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:56.607283 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.607266 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-credential-socket\") pod \"7ec938ed-e85d-42fe-a99c-301a2b361fef\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " Oct 31 03:48:56.607348 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.607330 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec938ed-e85d-42fe-a99c-301a2b361fef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec938ed-e85d-42fe-a99c-301a2b361fef-istiod-ca-cert\") pod \"7ec938ed-e85d-42fe-a99c-301a2b361fef\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " Oct 31 03:48:56.607390 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.607378 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-certs\") pod \"7ec938ed-e85d-42fe-a99c-301a2b361fef\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " Oct 31 03:48:56.607425 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.607399 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-podinfo\") pod \"7ec938ed-e85d-42fe-a99c-301a2b361fef\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " Oct 31 03:48:56.607525 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.607503 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-credential-socket" pod "7ec938ed-e85d-42fe-a99c-301a2b361fef" (UID: "7ec938ed-e85d-42fe-a99c-301a2b361fef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:56.607606 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.607515 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-token\" (UniqueName: \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-token\") pod \"7ec938ed-e85d-42fe-a99c-301a2b361fef\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " Oct 31 03:48:56.607606 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.607567 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-certs" pod "7ec938ed-e85d-42fe-a99c-301a2b361fef" (UID: "7ec938ed-e85d-42fe-a99c-301a2b361fef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:56.607705 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.607636 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-data\") pod \"7ec938ed-e85d-42fe-a99c-301a2b361fef\" (UID: \"7ec938ed-e85d-42fe-a99c-301a2b361fef\") " Oct 31 03:48:56.607705 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.607659 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec938ed-e85d-42fe-a99c-301a2b361fef-istiod-ca-cert" pod "7ec938ed-e85d-42fe-a99c-301a2b361fef" (UID: "7ec938ed-e85d-42fe-a99c-301a2b361fef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:56.608152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.607980 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-data" pod "7ec938ed-e85d-42fe-a99c-301a2b361fef" (UID: "7ec938ed-e85d-42fe-a99c-301a2b361fef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:56.608152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.608113 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:56.608152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.608153 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec938ed-e85d-42fe-a99c-301a2b361fef-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:56.608370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.608174 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:56.608370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.608187 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:56.608370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.608200 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:56.609928 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.609900 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-envoy" pod "7ec938ed-e85d-42fe-a99c-301a2b361fef" (UID: "7ec938ed-e85d-42fe-a99c-301a2b361fef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:56.610119 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.610093 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-token" pod "7ec938ed-e85d-42fe-a99c-301a2b361fef" (UID: "7ec938ed-e85d-42fe-a99c-301a2b361fef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:56.610238 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.610162 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-podinfo" pod "7ec938ed-e85d-42fe-a99c-301a2b361fef" (UID: "7ec938ed-e85d-42fe-a99c-301a2b361fef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:56.610779 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.610751 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-kube-api-access-qbvb8" pod "7ec938ed-e85d-42fe-a99c-301a2b361fef" (UID: "7ec938ed-e85d-42fe-a99c-301a2b361fef"). InnerVolumeSpecName "kube-api-access-qbvb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:56.709491 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.709460 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:56.709491 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.709489 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:56.709491 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.709498 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbvb8\" (UniqueName: \"kubernetes.io/projected/7ec938ed-e85d-42fe-a99c-301a2b361fef-kube-api-access-qbvb8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:56.709774 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:56.709507 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec938ed-e85d-42fe-a99c-301a2b361fef-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:57.110423 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:57.110344 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:48:57.110423 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:57.110397 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:48:57.213441 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:57.213405 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:48:57.213610 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:57.213459 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:48:57.496989 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:57.496948 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" event={"ID":"7ec938ed-e85d-42fe-a99c-301a2b361fef","Type":"ContainerDied","Data":"3e5551b52169a03cdae0a1e4876da5a1ef2596cc03ebad842ddc4c73b42cc960"} Oct 31 03:48:57.497192 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:57.497026 4426 scope.go:122] "RemoveContainer" containerID="6e020e444d920e041f3a7d6b6f5b9762633b95c7a6058311dd47526e6355ec47" Oct 31 03:48:57.497253 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:57.497195 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v" Oct 31 03:48:57.677390 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:57.677323 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v"] Oct 31 03:48:57.681801 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:57.681765 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wvs6v"] Oct 31 03:48:58.110363 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:58.110329 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:48:58.110543 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:58.110403 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:48:58.213148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:58.212676 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:48:58.213148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:58.212760 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:48:59.133699 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:59.133666 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" path="/var/lib/kubelet/pods/7ec938ed-e85d-42fe-a99c-301a2b361fef/volumes" Oct 31 03:48:59.134090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:59.134076 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:59.134419 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:59.134398 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:59.135621 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:59.135602 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 03:48:59.213209 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:59.213174 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:48:59.213409 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:48:59.213242 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:49:00.306937 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:00.306900 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:49:00.307474 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:00.307452 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:49:00.310546 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:00.310521 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 03:49:02.859458 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:49:02.859414 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:49:07.165451 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.165416 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:49:07.439348 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.439258 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7"] Oct 31 03:49:07.439830 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.439817 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" Oct 31 03:49:07.439875 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.439832 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" Oct 31 03:49:07.439959 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.439950 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec938ed-e85d-42fe-a99c-301a2b361fef" containerName="istio-proxy" Oct 31 03:49:07.445311 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.445282 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:07.494731 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.494696 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:49:07.494914 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.494753 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:49:07.498860 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.498830 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7"] Oct 31 03:49:07.522797 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.522762 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc8e788-nbzr7\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:07.522991 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.522859 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc8e788-nbzr7\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:07.523061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.522987 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfltc\" (UniqueName: \"kubernetes.io/projected/196fbf6e-b1f8-42ea-938f-2f1cde739855-kube-api-access-lfltc\") pod \"maas-api-revoke-sub-2cc8e788-nbzr7\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:07.543398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.543361 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:49:07.543398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.543391 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:49:07.634829 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.634782 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc8e788-nbzr7\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:07.635004 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.634916 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfltc\" (UniqueName: \"kubernetes.io/projected/196fbf6e-b1f8-42ea-938f-2f1cde739855-kube-api-access-lfltc\") pod \"maas-api-revoke-sub-2cc8e788-nbzr7\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:07.635004 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.634997 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc8e788-nbzr7\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:07.635850 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.635817 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc8e788-nbzr7\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:07.643212 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.640552 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc8e788-nbzr7\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:07.713198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.713079 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfltc\" (UniqueName: \"kubernetes.io/projected/196fbf6e-b1f8-42ea-938f-2f1cde739855-kube-api-access-lfltc\") pod \"maas-api-revoke-sub-2cc8e788-nbzr7\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:07.760412 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:07.760378 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:08.454688 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:49:08.440270 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196fbf6e_b1f8_42ea_938f_2f1cde739855.slice/crio-cc5c98601b1c56e2fb5bfcdbe201d3d76f28d840bf860319bf8e4ab278202360 WatchSource:0}: Error finding container cc5c98601b1c56e2fb5bfcdbe201d3d76f28d840bf860319bf8e4ab278202360: Status 404 returned error can't find the container with id cc5c98601b1c56e2fb5bfcdbe201d3d76f28d840bf860319bf8e4ab278202360 Oct 31 03:49:08.454688 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:08.443358 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7"] Oct 31 03:49:08.650151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:08.649757 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" event={"ID":"196fbf6e-b1f8-42ea-938f-2f1cde739855","Type":"ContainerStarted","Data":"cc5c98601b1c56e2fb5bfcdbe201d3d76f28d840bf860319bf8e4ab278202360"} Oct 31 03:49:09.659702 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:09.659656 4426 generic.go:356] "Generic (PLEG): container finished" podID="196fbf6e-b1f8-42ea-938f-2f1cde739855" containerID="e1ffa0e1818624d1ada84220029317cd1c7b909d021a36f3d49b13703c13d1d4" exitCode=0 Oct 31 03:49:09.660197 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:09.659750 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" event={"ID":"196fbf6e-b1f8-42ea-938f-2f1cde739855","Type":"ContainerDied","Data":"e1ffa0e1818624d1ada84220029317cd1c7b909d021a36f3d49b13703c13d1d4"} Oct 31 03:49:10.818559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:10.818534 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:10.905089 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:10.905057 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-xks-ca\") pod \"196fbf6e-b1f8-42ea-938f-2f1cde739855\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " Oct 31 03:49:10.905301 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:10.905193 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196fbf6e-b1f8-42ea-938f-2f1cde739855-kube-api-access-lfltc\" (UniqueName: \"kubernetes.io/projected/196fbf6e-b1f8-42ea-938f-2f1cde739855-kube-api-access-lfltc\") pod \"196fbf6e-b1f8-42ea-938f-2f1cde739855\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " Oct 31 03:49:10.905301 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:10.905284 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-openshift-ca\") pod \"196fbf6e-b1f8-42ea-938f-2f1cde739855\" (UID: \"196fbf6e-b1f8-42ea-938f-2f1cde739855\") " Oct 31 03:49:10.905628 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:10.905603 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-openshift-ca" pod "196fbf6e-b1f8-42ea-938f-2f1cde739855" (UID: "196fbf6e-b1f8-42ea-938f-2f1cde739855"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:49:10.907189 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:10.907158 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-xks-ca" pod "196fbf6e-b1f8-42ea-938f-2f1cde739855" (UID: "196fbf6e-b1f8-42ea-938f-2f1cde739855"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:49:10.907292 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:10.907219 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196fbf6e-b1f8-42ea-938f-2f1cde739855-kube-api-access-lfltc" pod "196fbf6e-b1f8-42ea-938f-2f1cde739855" (UID: "196fbf6e-b1f8-42ea-938f-2f1cde739855"). InnerVolumeSpecName "kube-api-access-lfltc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:11.006745 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:11.006655 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfltc\" (UniqueName: \"kubernetes.io/projected/196fbf6e-b1f8-42ea-938f-2f1cde739855-kube-api-access-lfltc\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:11.006745 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:11.006688 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:11.006745 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:11.006699 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196fbf6e-b1f8-42ea-938f-2f1cde739855-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:11.685603 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:11.685571 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" Oct 31 03:49:11.685603 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:11.685597 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7" event={"ID":"196fbf6e-b1f8-42ea-938f-2f1cde739855","Type":"ContainerDied","Data":"cc5c98601b1c56e2fb5bfcdbe201d3d76f28d840bf860319bf8e4ab278202360"} Oct 31 03:49:11.685808 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:11.685654 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc5c98601b1c56e2fb5bfcdbe201d3d76f28d840bf860319bf8e4ab278202360" Oct 31 03:49:26.838879 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.838843 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95"] Oct 31 03:49:26.839455 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.839406 4426 cpu_manager.go:414] "containerMap: removing container" podUID="196fbf6e-b1f8-42ea-938f-2f1cde739855" containerName="revoke-keys" Oct 31 03:49:26.839455 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.839456 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196fbf6e-b1f8-42ea-938f-2f1cde739855" containerName="revoke-keys" Oct 31 03:49:26.839626 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.839611 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="196fbf6e-b1f8-42ea-938f-2f1cde739855" containerName="revoke-keys" Oct 31 03:49:26.844898 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.844868 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:26.899448 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.899412 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95"] Oct 31 03:49:26.940168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.940109 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68aa3e\"/\"distinct-w4-68aa3e-kserve-self-signed-certs\"" Oct 31 03:49:26.994552 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.994507 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-dshm\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:26.994906 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.994878 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-model-cache\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:26.995210 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.995183 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-tmp-dir\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:26.995516 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.995488 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llmf5\" (UniqueName: \"kubernetes.io/projected/38467302-482f-4d7a-a23d-a52d5bf56c44-kube-api-access-llmf5\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:26.995766 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.995640 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38467302-482f-4d7a-a23d-a52d5bf56c44-tls-certs\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:26.995766 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:26.995733 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-home\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.097184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.097051 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-home\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.097184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.097111 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-dshm\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.097432 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.097200 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-model-cache\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.097432 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.097222 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-tmp-dir\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.097432 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.097283 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llmf5\" (UniqueName: \"kubernetes.io/projected/38467302-482f-4d7a-a23d-a52d5bf56c44-kube-api-access-llmf5\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.097432 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.097376 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38467302-482f-4d7a-a23d-a52d5bf56c44-tls-certs\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.097633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.097500 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-home\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.097633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.097610 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-model-cache\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.097707 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.097687 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-tmp-dir\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.099382 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.099358 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-dshm\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.099848 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.099826 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38467302-482f-4d7a-a23d-a52d5bf56c44-tls-certs\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.154924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.154885 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llmf5\" (UniqueName: \"kubernetes.io/projected/38467302-482f-4d7a-a23d-a52d5bf56c44-kube-api-access-llmf5\") pod \"distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.160787 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.160760 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:27.577105 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:49:27.577065 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38467302_482f_4d7a_a23d_a52d5bf56c44.slice/crio-341c22da841854f449e82d58784bbd7571051adafb1bf5817743998ed2c02b0d WatchSource:0}: Error finding container 341c22da841854f449e82d58784bbd7571051adafb1bf5817743998ed2c02b0d: Status 404 returned error can't find the container with id 341c22da841854f449e82d58784bbd7571051adafb1bf5817743998ed2c02b0d Oct 31 03:49:27.577999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.577971 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95"] Oct 31 03:49:27.838223 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.838109 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" event={"ID":"38467302-482f-4d7a-a23d-a52d5bf56c44","Type":"ContainerStarted","Data":"f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75"} Oct 31 03:49:27.838223 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.838179 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" event={"ID":"38467302-482f-4d7a-a23d-a52d5bf56c44","Type":"ContainerStarted","Data":"341c22da841854f449e82d58784bbd7571051adafb1bf5817743998ed2c02b0d"} Oct 31 03:49:27.950030 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:27.949967 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podStartSLOduration=1.949950429 podStartE2EDuration="1.949950429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:27.948517326 +0000 UTC m=+1264.949142555" watchObservedRunningTime="2026-10-31 03:49:27.949950429 +0000 UTC m=+1264.950575658" Oct 31 03:49:28.161251 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:28.161212 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:32.846074 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:49:32.846037 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:49:33.601586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.601553 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj"] Oct 31 03:49:33.606040 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.606020 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.702381 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.702342 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d7b7bc\"/\"distinct-w1-d7b7bc-kserve-self-signed-certs\"" Oct 31 03:49:33.707842 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.707813 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj"] Oct 31 03:49:33.778675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.778624 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-tmp-dir\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.778904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.778825 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s56l5\" (UniqueName: \"kubernetes.io/projected/954dfc88-35b6-4f05-b6c3-d083322ff454-kube-api-access-s56l5\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.778904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.778889 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954dfc88-35b6-4f05-b6c3-d083322ff454-tls-certs\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.779090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.779014 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-dshm\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.779191 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.779111 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-home\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.779252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.779201 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-model-cache\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.879973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.879879 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s56l5\" (UniqueName: \"kubernetes.io/projected/954dfc88-35b6-4f05-b6c3-d083322ff454-kube-api-access-s56l5\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.879973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.879929 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954dfc88-35b6-4f05-b6c3-d083322ff454-tls-certs\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.879973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.879955 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-dshm\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.880590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.880006 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-home\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.880590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.880028 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-model-cache\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.880590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.880113 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-tmp-dir\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.880590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.880523 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-home\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.880590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.880556 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-model-cache\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.880851 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.880618 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-tmp-dir\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.882575 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.882546 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-dshm\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.882731 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.882689 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954dfc88-35b6-4f05-b6c3-d083322ff454-tls-certs\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:33.938221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:33.938186 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s56l5\" (UniqueName: \"kubernetes.io/projected/954dfc88-35b6-4f05-b6c3-d083322ff454-kube-api-access-s56l5\") pod \"distinct-w1-d7b7bc-kserve-7b668698fc-p4msj\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:34.221204 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:34.221165 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:34.599185 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:34.599153 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj"] Oct 31 03:49:34.599315 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:49:34.599253 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod954dfc88_35b6_4f05_b6c3_d083322ff454.slice/crio-4c27ac417a01cdd852db980b19a0c73c0fe8230407d5838d553983a60a604b26 WatchSource:0}: Error finding container 4c27ac417a01cdd852db980b19a0c73c0fe8230407d5838d553983a60a604b26: Status 404 returned error can't find the container with id 4c27ac417a01cdd852db980b19a0c73c0fe8230407d5838d553983a60a604b26 Oct 31 03:49:34.904473 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:34.904429 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" event={"ID":"954dfc88-35b6-4f05-b6c3-d083322ff454","Type":"ContainerStarted","Data":"212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10"} Oct 31 03:49:34.905072 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:34.904490 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" event={"ID":"954dfc88-35b6-4f05-b6c3-d083322ff454","Type":"ContainerStarted","Data":"4c27ac417a01cdd852db980b19a0c73c0fe8230407d5838d553983a60a604b26"} Oct 31 03:49:35.022703 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:35.022648 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podStartSLOduration=2.02263161 podStartE2EDuration="2.02263161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:35.01987898 +0000 UTC m=+1272.020504208" watchObservedRunningTime="2026-10-31 03:49:35.02263161 +0000 UTC m=+1272.023256840" Oct 31 03:49:35.221844 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:35.221756 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:37.161565 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:37.161525 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:37.174531 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:37.174500 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:37.941674 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:37.941644 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:49:41.137295 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:41.137256 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:49:44.221513 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:44.221473 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:44.239764 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:44.239728 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:49:45.066843 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:49:45.066810 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:50:02.726242 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.726200 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt"] Oct 31 03:50:02.731989 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.731937 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.779501 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.779463 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt"] Oct 31 03:50:02.785099 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.785055 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-412ba5\"/\"kube-root-ca.crt\"" Oct 31 03:50:02.796846 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.796809 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qnz5\" (UniqueName: \"kubernetes.io/projected/5072cd0c-fc68-4271-96a5-71fd809cf75b-kube-api-access-5qnz5\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.797052 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.796863 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-tmp-dir\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.797052 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.796998 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5072cd0c-fc68-4271-96a5-71fd809cf75b-tls-certs\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.797195 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.797053 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-dshm\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.797195 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.797074 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-model-cache\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.797302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.797209 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-home\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.822921 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.822885 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-412ba5\"/\"default-dockercfg-rq9bv\"" Oct 31 03:50:02.824437 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.823261 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-412ba5\"/\"openshift-service-ca.crt\"" Oct 31 03:50:02.824648 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.823367 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-412ba5\"/\"distinct-2-w6-412ba5-kserve-self-signed-certs\"" Oct 31 03:50:02.895171 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:50:02.893329 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:50:02.898093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.898060 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-dshm\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.898263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.898105 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-model-cache\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.898263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.898174 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-home\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.898365 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.898332 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qnz5\" (UniqueName: \"kubernetes.io/projected/5072cd0c-fc68-4271-96a5-71fd809cf75b-kube-api-access-5qnz5\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.898417 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.898382 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-tmp-dir\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.898487 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.898434 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5072cd0c-fc68-4271-96a5-71fd809cf75b-tls-certs\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.898607 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.898582 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-model-cache\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.898696 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.898622 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-home\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.903160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.898852 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-tmp-dir\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.903160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.900610 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-dshm\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.903160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.901325 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5072cd0c-fc68-4271-96a5-71fd809cf75b-tls-certs\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:02.956398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:02.956359 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qnz5\" (UniqueName: \"kubernetes.io/projected/5072cd0c-fc68-4271-96a5-71fd809cf75b-kube-api-access-5qnz5\") pod \"distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:03.064021 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:03.062713 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:03.478686 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:03.478655 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt"] Oct 31 03:50:03.480337 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:50:03.480302 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5072cd0c_fc68_4271_96a5_71fd809cf75b.slice/crio-c4049c582982ded3e7c90eb860dd5340f334370a22421bd9082a8370cf30ed24 WatchSource:0}: Error finding container c4049c582982ded3e7c90eb860dd5340f334370a22421bd9082a8370cf30ed24: Status 404 returned error can't find the container with id c4049c582982ded3e7c90eb860dd5340f334370a22421bd9082a8370cf30ed24 Oct 31 03:50:04.226149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:04.222241 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" event={"ID":"5072cd0c-fc68-4271-96a5-71fd809cf75b","Type":"ContainerStarted","Data":"e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4"} Oct 31 03:50:04.226149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:04.222318 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" event={"ID":"5072cd0c-fc68-4271-96a5-71fd809cf75b","Type":"ContainerStarted","Data":"c4049c582982ded3e7c90eb860dd5340f334370a22421bd9082a8370cf30ed24"} Oct 31 03:50:04.336957 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:04.336900 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podStartSLOduration=2.336884944 podStartE2EDuration="2.336884944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:04.333689074 +0000 UTC m=+1301.334314296" watchObservedRunningTime="2026-10-31 03:50:04.336884944 +0000 UTC m=+1301.337510172" Oct 31 03:50:05.063892 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:05.063856 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:07.258266 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:07.257149 4426 generic.go:356] "Generic (PLEG): container finished" podID="507c0207-cbfd-48a6-b463-5e7b615589b6" containerID="dd0614a4b3a1d5647c34ecb0e33dc8ff04bf57658820b537c886aa3f3d61161d" exitCode=0 Oct 31 03:50:07.258266 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:07.257343 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" event={"ID":"507c0207-cbfd-48a6-b463-5e7b615589b6","Type":"ContainerDied","Data":"dd0614a4b3a1d5647c34ecb0e33dc8ff04bf57658820b537c886aa3f3d61161d"} Oct 31 03:50:08.418818 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.418791 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:50:08.487200 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.487121 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507c0207-cbfd-48a6-b463-5e7b615589b6-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/507c0207-cbfd-48a6-b463-5e7b615589b6-archives-path\") pod \"507c0207-cbfd-48a6-b463-5e7b615589b6\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " Oct 31 03:50:08.487391 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.487235 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-trusted-ca-bundle\") pod \"507c0207-cbfd-48a6-b463-5e7b615589b6\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " Oct 31 03:50:08.487391 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.487314 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-service-ca-bundle\") pod \"507c0207-cbfd-48a6-b463-5e7b615589b6\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " Oct 31 03:50:08.487391 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.487363 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507c0207-cbfd-48a6-b463-5e7b615589b6-kube-api-access-zndc8\" (UniqueName: \"kubernetes.io/projected/507c0207-cbfd-48a6-b463-5e7b615589b6-kube-api-access-zndc8\") pod \"507c0207-cbfd-48a6-b463-5e7b615589b6\" (UID: \"507c0207-cbfd-48a6-b463-5e7b615589b6\") " Oct 31 03:50:08.487520 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.487488 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507c0207-cbfd-48a6-b463-5e7b615589b6-archives-path" pod "507c0207-cbfd-48a6-b463-5e7b615589b6" (UID: "507c0207-cbfd-48a6-b463-5e7b615589b6"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:08.487641 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.487621 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-trusted-ca-bundle" pod "507c0207-cbfd-48a6-b463-5e7b615589b6" (UID: "507c0207-cbfd-48a6-b463-5e7b615589b6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:08.487684 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.487661 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-service-ca-bundle" pod "507c0207-cbfd-48a6-b463-5e7b615589b6" (UID: "507c0207-cbfd-48a6-b463-5e7b615589b6"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:08.487852 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.487835 4426 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/507c0207-cbfd-48a6-b463-5e7b615589b6-archives-path\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:08.487885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.487854 4426 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-trusted-ca-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:08.487885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.487864 4426 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/507c0207-cbfd-48a6-b463-5e7b615589b6-service-ca-bundle\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:08.489472 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.489452 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507c0207-cbfd-48a6-b463-5e7b615589b6-kube-api-access-zndc8" pod "507c0207-cbfd-48a6-b463-5e7b615589b6" (UID: "507c0207-cbfd-48a6-b463-5e7b615589b6"). InnerVolumeSpecName "kube-api-access-zndc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:08.588723 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:08.588623 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zndc8\" (UniqueName: \"kubernetes.io/projected/507c0207-cbfd-48a6-b463-5e7b615589b6-kube-api-access-zndc8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:09.274754 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:09.274721 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" Oct 31 03:50:09.274754 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:09.274746 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wr6bj-n6rpw" event={"ID":"507c0207-cbfd-48a6-b463-5e7b615589b6","Type":"ContainerDied","Data":"ad08440b938a6290d6db2c67745a66d49c4666638d3f9c70ac1ba28fa6098030"} Oct 31 03:50:09.275004 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:09.274821 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad08440b938a6290d6db2c67745a66d49c4666638d3f9c70ac1ba28fa6098030" Oct 31 03:50:13.063708 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:13.063672 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:13.076826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:13.076795 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:13.320649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:13.320571 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:50:20.573089 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:20.573044 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:50:32.847724 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:50:32.847680 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:50:37.672600 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.672564 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs"] Oct 31 03:50:37.673157 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.673137 4426 cpu_manager.go:414] "containerMap: removing container" podUID="507c0207-cbfd-48a6-b463-5e7b615589b6" containerName="insights-gathering" Oct 31 03:50:37.673157 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.673152 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507c0207-cbfd-48a6-b463-5e7b615589b6" containerName="insights-gathering" Oct 31 03:50:37.673284 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.673274 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="507c0207-cbfd-48a6-b463-5e7b615589b6" containerName="insights-gathering" Oct 31 03:50:37.678160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.678107 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.762467 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.762429 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-data\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.762655 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.762475 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-credential-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.762739 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.762680 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61524595-92d6-418c-ba38-9411c584464b-istio-podinfo\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.762739 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.762724 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.762853 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.762790 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-certs\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.762853 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.762825 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-istio-token\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.762962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.762869 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-envoy\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.762962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.762896 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61524595-92d6-418c-ba38-9411c584464b-istiod-ca-cert\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.762962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.762932 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6gs5\" (UniqueName: \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-kube-api-access-d6gs5\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.864260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864208 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-data\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.864260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864267 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-credential-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.864520 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864394 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61524595-92d6-418c-ba38-9411c584464b-istio-podinfo\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.864520 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864429 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.864520 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864451 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-certs\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.864520 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864483 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-istio-token\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.864734 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864529 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-envoy\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.864734 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864556 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61524595-92d6-418c-ba38-9411c584464b-istiod-ca-cert\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.864734 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864593 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6gs5\" (UniqueName: \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-kube-api-access-d6gs5\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.864899 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864726 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-data\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.864899 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864761 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-credential-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.865031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864916 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-certs\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.865031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.864933 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-socket\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.865514 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.865487 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61524595-92d6-418c-ba38-9411c584464b-istiod-ca-cert\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.867149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.867115 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61524595-92d6-418c-ba38-9411c584464b-istio-podinfo\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.867312 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.867271 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-envoy\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.890312 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.890279 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs"] Oct 31 03:50:37.922484 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.922449 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-istio-token\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.922684 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.922548 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6gs5\" (UniqueName: \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-kube-api-access-d6gs5\") pod \"e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:37.992890 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:37.992851 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:38.421614 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:50:38.421582 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61524595_92d6_418c_ba38_9411c584464b.slice/crio-6736a21a5cf6a7484b4e38061e15415dbc3ae0520c1e68b3d73ef5b729a071c4 WatchSource:0}: Error finding container 6736a21a5cf6a7484b4e38061e15415dbc3ae0520c1e68b3d73ef5b729a071c4: Status 404 returned error can't find the container with id 6736a21a5cf6a7484b4e38061e15415dbc3ae0520c1e68b3d73ef5b729a071c4 Oct 31 03:50:38.422100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:38.422074 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs"] Oct 31 03:50:38.424025 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:38.423992 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:50:38.424113 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:38.424057 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:50:38.424113 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:38.424087 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:50:38.546418 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:38.546378 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" event={"ID":"61524595-92d6-418c-ba38-9411c584464b","Type":"ContainerStarted","Data":"122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341"} Oct 31 03:50:38.546586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:38.546431 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" event={"ID":"61524595-92d6-418c-ba38-9411c584464b","Type":"ContainerStarted","Data":"6736a21a5cf6a7484b4e38061e15415dbc3ae0520c1e68b3d73ef5b729a071c4"} Oct 31 03:50:38.663100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:38.663018 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" podStartSLOduration=1.663002453 podStartE2EDuration="1.663002453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:38.662373872 +0000 UTC m=+1335.662999101" watchObservedRunningTime="2026-10-31 03:50:38.663002453 +0000 UTC m=+1335.663627682" Oct 31 03:50:38.993740 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:38.993701 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:39.995027 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:39.994989 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:50:39.995548 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:39.995044 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:50:40.993616 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:40.993578 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:50:40.993798 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:40.993654 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:50:41.993322 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:41.993280 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:50:41.993737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:41.993342 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:50:42.993588 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:42.993548 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:50:42.994011 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:42.993620 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:50:43.994137 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:43.994079 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:50:43.994560 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:43.994192 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:50:44.994154 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:44.994100 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:50:44.994551 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:44.994175 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:50:45.997979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:45.997949 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:45.998497 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:45.998484 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:50:45.999080 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:50:45.999057 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 03:51:02.846409 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:51:02.846369 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:51:07.175401 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:07.175363 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh"] Oct 31 03:51:07.175804 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:07.175575 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" containerID="cri-o://a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f" gracePeriod=30 Oct 31 03:51:08.142380 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.142344 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd"] Oct 31 03:51:08.147062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.147030 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:08.217502 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.217460 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd"] Oct 31 03:51:08.242255 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.242219 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:51:08.242255 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.242225 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:51:08.242552 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.242313 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:51:08.242749 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.242722 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:51:08.269343 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.269300 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4d9\" (UniqueName: \"kubernetes.io/projected/ca520e83-5d3e-4293-9312-de39429cd6fc-kube-api-access-6p4d9\") pod \"maas-api-revoke-sub-a49e9114-qhwjd\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:08.269538 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.269472 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49e9114-qhwjd\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:08.269608 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.269542 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49e9114-qhwjd\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:08.370765 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.370726 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4d9\" (UniqueName: \"kubernetes.io/projected/ca520e83-5d3e-4293-9312-de39429cd6fc-kube-api-access-6p4d9\") pod \"maas-api-revoke-sub-a49e9114-qhwjd\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:08.370984 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.370899 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49e9114-qhwjd\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:08.370984 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.370949 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49e9114-qhwjd\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:08.371633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.371597 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49e9114-qhwjd\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:08.373384 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.373362 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49e9114-qhwjd\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:08.428855 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.428766 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4d9\" (UniqueName: \"kubernetes.io/projected/ca520e83-5d3e-4293-9312-de39429cd6fc-kube-api-access-6p4d9\") pod \"maas-api-revoke-sub-a49e9114-qhwjd\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:08.467554 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.467515 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:08.850670 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:08.850640 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd"] Oct 31 03:51:08.851329 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:51:08.851299 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca520e83_5d3e_4293_9312_de39429cd6fc.slice/crio-c9eb9466a1cafdc657cd01a070fabbf075b692ae30d953383c81e0f93c3489ff WatchSource:0}: Error finding container c9eb9466a1cafdc657cd01a070fabbf075b692ae30d953383c81e0f93c3489ff: Status 404 returned error can't find the container with id c9eb9466a1cafdc657cd01a070fabbf075b692ae30d953383c81e0f93c3489ff Oct 31 03:51:09.820405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:09.820361 4426 generic.go:356] "Generic (PLEG): container finished" podID="ca520e83-5d3e-4293-9312-de39429cd6fc" containerID="aedbcada780fb8ef42b4d8a8615bc26a05508c1847e69566d7f9213e2949c703" exitCode=0 Oct 31 03:51:09.820848 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:09.820427 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" event={"ID":"ca520e83-5d3e-4293-9312-de39429cd6fc","Type":"ContainerDied","Data":"aedbcada780fb8ef42b4d8a8615bc26a05508c1847e69566d7f9213e2949c703"} Oct 31 03:51:09.820848 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:09.820483 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" event={"ID":"ca520e83-5d3e-4293-9312-de39429cd6fc","Type":"ContainerStarted","Data":"c9eb9466a1cafdc657cd01a070fabbf075b692ae30d953383c81e0f93c3489ff"} Oct 31 03:51:10.989025 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:10.989000 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:11.106227 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.105766 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-openshift-ca\") pod \"ca520e83-5d3e-4293-9312-de39429cd6fc\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " Oct 31 03:51:11.106227 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.105867 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-xks-ca\") pod \"ca520e83-5d3e-4293-9312-de39429cd6fc\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " Oct 31 03:51:11.106227 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.105987 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca520e83-5d3e-4293-9312-de39429cd6fc-kube-api-access-6p4d9\" (UniqueName: \"kubernetes.io/projected/ca520e83-5d3e-4293-9312-de39429cd6fc-kube-api-access-6p4d9\") pod \"ca520e83-5d3e-4293-9312-de39429cd6fc\" (UID: \"ca520e83-5d3e-4293-9312-de39429cd6fc\") " Oct 31 03:51:11.114594 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.114556 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca520e83-5d3e-4293-9312-de39429cd6fc-kube-api-access-6p4d9" pod "ca520e83-5d3e-4293-9312-de39429cd6fc" (UID: "ca520e83-5d3e-4293-9312-de39429cd6fc"). InnerVolumeSpecName "kube-api-access-6p4d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:11.114929 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.114905 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-openshift-ca" pod "ca520e83-5d3e-4293-9312-de39429cd6fc" (UID: "ca520e83-5d3e-4293-9312-de39429cd6fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:51:11.116894 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.116862 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-xks-ca" pod "ca520e83-5d3e-4293-9312-de39429cd6fc" (UID: "ca520e83-5d3e-4293-9312-de39429cd6fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:51:11.207900 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.207849 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:11.207900 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.207895 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca520e83-5d3e-4293-9312-de39429cd6fc-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:11.207900 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.207908 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p4d9\" (UniqueName: \"kubernetes.io/projected/ca520e83-5d3e-4293-9312-de39429cd6fc-kube-api-access-6p4d9\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:11.459766 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.459726 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:51:11.840211 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.840092 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" Oct 31 03:51:11.840211 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.840150 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd" event={"ID":"ca520e83-5d3e-4293-9312-de39429cd6fc","Type":"ContainerDied","Data":"c9eb9466a1cafdc657cd01a070fabbf075b692ae30d953383c81e0f93c3489ff"} Oct 31 03:51:11.840451 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:11.840211 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9eb9466a1cafdc657cd01a070fabbf075b692ae30d953383c81e0f93c3489ff" Oct 31 03:51:12.464669 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.464591 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:51:12.630038 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.629999 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-credential-socket\") pod \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " Oct 31 03:51:12.630266 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.630056 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-envoy\") pod \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " Oct 31 03:51:12.630266 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.630102 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istiod-ca-cert\") pod \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " Oct 31 03:51:12.630266 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.630227 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-certs\") pod \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " Oct 31 03:51:12.630518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.630303 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-socket\") pod \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " Oct 31 03:51:12.630518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.630335 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-kube-api-access-sf6t4\" (UniqueName: \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-kube-api-access-sf6t4\") pod \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " Oct 31 03:51:12.630518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.630412 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-podinfo\") pod \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " Oct 31 03:51:12.630518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.630474 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-token\" (UniqueName: \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-token\") pod \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " Oct 31 03:51:12.630744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.630538 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-data\") pod \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\" (UID: \"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea\") " Oct 31 03:51:12.631422 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.631086 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-socket" pod "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" (UID: "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:12.631422 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.631305 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-data" pod "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" (UID: "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:12.631641 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.631417 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-credential-socket" pod "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" (UID: "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:12.631858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.631732 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istiod-ca-cert" pod "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" (UID: "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:51:12.631858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.631818 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-certs" pod "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" (UID: "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:12.638174 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.634520 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-envoy" pod "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" (UID: "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:12.638174 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.634616 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-podinfo" pod "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" (UID: "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:51:12.664221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.651929 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-token" pod "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" (UID: "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:12.671080 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.671021 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-kube-api-access-sf6t4" pod "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" (UID: "c41b032c-a9c2-4aac-b4d9-9d87560ba1ea"). InnerVolumeSpecName "kube-api-access-sf6t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:12.732485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.732378 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:12.732485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.732421 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:12.732485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.732437 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:12.732485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.732452 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:12.732485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.732466 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:12.732485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.732480 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:12.732962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.732495 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:12.732962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.732511 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf6t4\" (UniqueName: \"kubernetes.io/projected/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-kube-api-access-sf6t4\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:12.732962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.732528 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:12.854587 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.854544 4426 generic.go:356] "Generic (PLEG): container finished" podID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerID="a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f" exitCode=0 Oct 31 03:51:12.854800 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.854610 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" event={"ID":"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea","Type":"ContainerDied","Data":"a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f"} Oct 31 03:51:12.854800 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.854693 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" event={"ID":"c41b032c-a9c2-4aac-b4d9-9d87560ba1ea","Type":"ContainerDied","Data":"d8e57b70191b4aa2d122c2de977349850f3286e7db93b68aa3f73766fb5093e0"} Oct 31 03:51:12.854800 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.854724 4426 scope.go:122] "RemoveContainer" containerID="a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f" Oct 31 03:51:12.854985 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.854903 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh" Oct 31 03:51:12.873043 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.872997 4426 scope.go:122] "RemoveContainer" containerID="7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b" Oct 31 03:51:12.889527 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.889504 4426 scope.go:122] "RemoveContainer" containerID="a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f" Oct 31 03:51:12.890031 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:51:12.890007 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f\": container with ID starting with a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f not found: ID does not exist" containerID="a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f" Oct 31 03:51:12.890092 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.890043 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f"} err="failed to get container status \"a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f\": rpc error: code = NotFound desc = could not find container \"a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f\": container with ID starting with a865179786899d62468935f34f0278dc8951c22a0fa1f268cee621f8680e432f not found: ID does not exist" Oct 31 03:51:12.890092 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.890065 4426 scope.go:122] "RemoveContainer" containerID="7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b" Oct 31 03:51:12.890401 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:51:12.890377 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b\": container with ID starting with 7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b not found: ID does not exist" containerID="7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b" Oct 31 03:51:12.890468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:12.890408 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b"} err="failed to get container status \"7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b\": rpc error: code = NotFound desc = could not find container \"7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b\": container with ID starting with 7ef9cfb5632d24995b59ceb15428da5955b1696fef92fdac3f034a586bcab93b not found: ID does not exist" Oct 31 03:51:13.013985 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:13.013897 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh"] Oct 31 03:51:13.023893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:13.023861 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-kjrmh"] Oct 31 03:51:13.132759 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:13.132724 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" path="/var/lib/kubelet/pods/c41b032c-a9c2-4aac-b4d9-9d87560ba1ea/volumes" Oct 31 03:51:27.046597 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.046559 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4"] Oct 31 03:51:27.050155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.048722 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" Oct 31 03:51:27.050155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.048757 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" Oct 31 03:51:27.050155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.048800 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" Oct 31 03:51:27.050155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.048808 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" Oct 31 03:51:27.050155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.048919 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ca520e83-5d3e-4293-9312-de39429cd6fc" containerName="revoke-keys" Oct 31 03:51:27.050155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.048927 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca520e83-5d3e-4293-9312-de39429cd6fc" containerName="revoke-keys" Oct 31 03:51:27.050155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.049211 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" Oct 31 03:51:27.050155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.049244 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca520e83-5d3e-4293-9312-de39429cd6fc" containerName="revoke-keys" Oct 31 03:51:27.058228 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.057091 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.104599 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.104562 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-home\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.104809 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.104636 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-tmp-dir\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.104809 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.104657 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zjkm\" (UniqueName: \"kubernetes.io/projected/3679293e-d231-4dd6-b1c5-3409a505af6c-kube-api-access-2zjkm\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.104809 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.104737 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-model-cache\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.104982 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.104868 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-dshm\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.104982 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.104919 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3679293e-d231-4dd6-b1c5-3409a505af6c-tls-certs\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.106978 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.106951 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4"] Oct 31 03:51:27.147531 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.147501 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d7b7bc\"/\"distinct-2-w1-d7b7bc-kserve-self-signed-certs\"" Oct 31 03:51:27.206559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.206441 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-model-cache\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.206908 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.206592 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-dshm\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.206908 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.206647 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3679293e-d231-4dd6-b1c5-3409a505af6c-tls-certs\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.206908 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.206699 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-home\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.206908 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.206768 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-tmp-dir\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.207236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.207206 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zjkm\" (UniqueName: \"kubernetes.io/projected/3679293e-d231-4dd6-b1c5-3409a505af6c-kube-api-access-2zjkm\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.207411 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.207384 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-tmp-dir\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.207689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.207668 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-model-cache\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.209865 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.209834 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-home\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.210755 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.210732 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-dshm\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.211706 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.211682 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3679293e-d231-4dd6-b1c5-3409a505af6c-tls-certs\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.265022 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.264986 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zjkm\" (UniqueName: \"kubernetes.io/projected/3679293e-d231-4dd6-b1c5-3409a505af6c-kube-api-access-2zjkm\") pod \"distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.372085 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.371998 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:27.758717 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:51:27.758677 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3679293e_d231_4dd6_b1c5_3409a505af6c.slice/crio-d2858785717976d19628ed297521614750e75dcd65394312295959bd5016669c WatchSource:0}: Error finding container d2858785717976d19628ed297521614750e75dcd65394312295959bd5016669c: Status 404 returned error can't find the container with id d2858785717976d19628ed297521614750e75dcd65394312295959bd5016669c Oct 31 03:51:27.758893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.758873 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4"] Oct 31 03:51:27.986109 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.986069 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" event={"ID":"3679293e-d231-4dd6-b1c5-3409a505af6c","Type":"ContainerStarted","Data":"1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32"} Oct 31 03:51:27.986296 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:27.986120 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" event={"ID":"3679293e-d231-4dd6-b1c5-3409a505af6c","Type":"ContainerStarted","Data":"d2858785717976d19628ed297521614750e75dcd65394312295959bd5016669c"} Oct 31 03:51:28.100170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:28.100052 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podStartSLOduration=1.10003787 podStartE2EDuration="1.10003787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:28.09502519 +0000 UTC m=+1385.095650419" watchObservedRunningTime="2026-10-31 03:51:28.10003787 +0000 UTC m=+1385.100663117" Oct 31 03:51:28.372648 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:28.372561 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:32.061788 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:32.061748 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2"] Oct 31 03:51:32.107478 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:32.107438 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cec0039-9ztt2"] Oct 31 03:51:32.846979 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:51:32.846935 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:51:33.145485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:33.145213 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a4c766-3923-4dd7-84bc-b338430976d4" path="/var/lib/kubelet/pods/c7a4c766-3923-4dd7-84bc-b338430976d4/volumes" Oct 31 03:51:37.372311 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:37.372227 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:37.385654 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:37.385627 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:38.093544 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:38.093514 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:51:48.048755 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:48.048715 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn"] Oct 31 03:51:48.095172 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:48.095115 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6"] Oct 31 03:51:48.099518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:48.099487 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353a4c3c-4bksn"] Oct 31 03:51:48.103598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:48.103575 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ca26021-9djp6"] Oct 31 03:51:49.131208 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:49.131172 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c94bd5-1985-42be-ae63-0004dfcddfde" path="/var/lib/kubelet/pods/02c94bd5-1985-42be-ae63-0004dfcddfde/volumes" Oct 31 03:51:49.131864 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:49.131844 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc49eec-476b-4c81-8d47-9f14cc4ce77c" path="/var/lib/kubelet/pods/bbc49eec-476b-4c81-8d47-9f14cc4ce77c/volumes" Oct 31 03:51:49.616643 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:49.616609 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:51:56.549009 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.548971 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk"] Oct 31 03:51:56.550027 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.550004 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41b032c-a9c2-4aac-b4d9-9d87560ba1ea" containerName="istio-proxy" Oct 31 03:51:56.555413 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.555391 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.558957 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.558932 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33640080-5615-4d98-bbcb-83fa262c3fe6-tls-certs\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.559082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.558991 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-dshm\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.559232 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.559163 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-tmp-dir\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.559330 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.559306 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-home\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.559382 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.559358 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-model-cache\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.559492 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.559474 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5bfx\" (UniqueName: \"kubernetes.io/projected/33640080-5615-4d98-bbcb-83fa262c3fe6-kube-api-access-g5bfx\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.608112 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.608077 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk"] Oct 31 03:51:56.647658 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.647620 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68aa3e\"/\"distinct-2-w4-68aa3e-kserve-self-signed-certs\"" Oct 31 03:51:56.660357 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.660322 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-tmp-dir\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.662259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.660428 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-home\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.662259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.660465 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-model-cache\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.662259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.660525 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5bfx\" (UniqueName: \"kubernetes.io/projected/33640080-5615-4d98-bbcb-83fa262c3fe6-kube-api-access-g5bfx\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.662259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.660602 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33640080-5615-4d98-bbcb-83fa262c3fe6-tls-certs\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.662259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.660661 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-dshm\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.662259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.660811 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-tmp-dir\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.662259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.660854 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-home\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.662259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.661531 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-model-cache\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.663231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.663209 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-dshm\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.663725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.663705 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33640080-5615-4d98-bbcb-83fa262c3fe6-tls-certs\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.720430 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.720380 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5bfx\" (UniqueName: \"kubernetes.io/projected/33640080-5615-4d98-bbcb-83fa262c3fe6-kube-api-access-g5bfx\") pod \"distinct-2-w4-68aa3e-kserve-67949648bb-7vglk\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:56.871700 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:56.871600 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:51:57.251394 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:57.251357 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk"] Oct 31 03:51:57.251773 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:51:57.251744 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33640080_5615_4d98_bbcb_83fa262c3fe6.slice/crio-fbeeae1aa1dd3be91a8918157a6a14d129fd45c4a6b497aff7cb29c9c85828e0 WatchSource:0}: Error finding container fbeeae1aa1dd3be91a8918157a6a14d129fd45c4a6b497aff7cb29c9c85828e0: Status 404 returned error can't find the container with id fbeeae1aa1dd3be91a8918157a6a14d129fd45c4a6b497aff7cb29c9c85828e0 Oct 31 03:51:58.268062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:58.268023 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" event={"ID":"33640080-5615-4d98-bbcb-83fa262c3fe6","Type":"ContainerStarted","Data":"fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5"} Oct 31 03:51:58.268486 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:58.268075 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" event={"ID":"33640080-5615-4d98-bbcb-83fa262c3fe6","Type":"ContainerStarted","Data":"fbeeae1aa1dd3be91a8918157a6a14d129fd45c4a6b497aff7cb29c9c85828e0"} Oct 31 03:51:58.382976 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:58.382920 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podStartSLOduration=2.3829041220000002 podStartE2EDuration="2.382904122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:58.377905612 +0000 UTC m=+1415.378530841" watchObservedRunningTime="2026-10-31 03:51:58.382904122 +0000 UTC m=+1415.383529350" Oct 31 03:51:58.872155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:51:58.872096 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:52:00.085146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.080872 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc"] Oct 31 03:52:00.088422 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.088394 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:00.130321 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.130286 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjqql\" (UniqueName: \"kubernetes.io/projected/5257abaf-67a0-45f6-80d2-077bc7e506d3-kube-api-access-jjqql\") pod \"maas-api-revoke-sub-2da14249-xvcdc\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:00.130491 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.130442 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2da14249-xvcdc\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:00.130572 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.130538 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2da14249-xvcdc\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:00.139590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.139564 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc"] Oct 31 03:52:00.182089 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.182051 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:52:00.182336 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.182102 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:52:00.182336 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.182207 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:52:00.184379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.184348 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:52:00.232063 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.232030 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjqql\" (UniqueName: \"kubernetes.io/projected/5257abaf-67a0-45f6-80d2-077bc7e506d3-kube-api-access-jjqql\") pod \"maas-api-revoke-sub-2da14249-xvcdc\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:00.232267 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.232112 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2da14249-xvcdc\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:00.232267 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.232150 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2da14249-xvcdc\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:00.233013 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.232976 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2da14249-xvcdc\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:00.234632 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.234600 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2da14249-xvcdc\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:00.309066 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.309026 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjqql\" (UniqueName: \"kubernetes.io/projected/5257abaf-67a0-45f6-80d2-077bc7e506d3-kube-api-access-jjqql\") pod \"maas-api-revoke-sub-2da14249-xvcdc\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:00.404765 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.404729 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:00.799083 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:52:00.799048 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5257abaf_67a0_45f6_80d2_077bc7e506d3.slice/crio-32aacc047e9f5fd928827a2bec7b86ae32c75a0173843e29d19f32b14e90f6e3 WatchSource:0}: Error finding container 32aacc047e9f5fd928827a2bec7b86ae32c75a0173843e29d19f32b14e90f6e3: Status 404 returned error can't find the container with id 32aacc047e9f5fd928827a2bec7b86ae32c75a0173843e29d19f32b14e90f6e3 Oct 31 03:52:00.799469 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:00.799448 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc"] Oct 31 03:52:01.297935 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:01.297891 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" event={"ID":"5257abaf-67a0-45f6-80d2-077bc7e506d3","Type":"ContainerStarted","Data":"ff048e83894f1aef6d13961d98ef6ef474b17bac893cfc7671fea18506e9a866"} Oct 31 03:52:01.298364 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:01.297952 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" event={"ID":"5257abaf-67a0-45f6-80d2-077bc7e506d3","Type":"ContainerStarted","Data":"32aacc047e9f5fd928827a2bec7b86ae32c75a0173843e29d19f32b14e90f6e3"} Oct 31 03:52:01.411024 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:01.410963 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" podStartSLOduration=1.41094833 podStartE2EDuration="1.41094833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:01.40645486 +0000 UTC m=+1418.407080090" watchObservedRunningTime="2026-10-31 03:52:01.41094833 +0000 UTC m=+1418.411573558" Oct 31 03:52:02.308682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:02.308648 4426 generic.go:356] "Generic (PLEG): container finished" podID="5257abaf-67a0-45f6-80d2-077bc7e506d3" containerID="ff048e83894f1aef6d13961d98ef6ef474b17bac893cfc7671fea18506e9a866" exitCode=0 Oct 31 03:52:02.309168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:02.308731 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" event={"ID":"5257abaf-67a0-45f6-80d2-077bc7e506d3","Type":"ContainerDied","Data":"ff048e83894f1aef6d13961d98ef6ef474b17bac893cfc7671fea18506e9a866"} Oct 31 03:52:02.848246 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:52:02.848204 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:52:03.492934 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:03.492900 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:03.581480 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:03.581391 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-openshift-ca\") pod \"5257abaf-67a0-45f6-80d2-077bc7e506d3\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " Oct 31 03:52:03.581480 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:03.581437 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-xks-ca\") pod \"5257abaf-67a0-45f6-80d2-077bc7e506d3\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " Oct 31 03:52:03.581675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:03.581495 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5257abaf-67a0-45f6-80d2-077bc7e506d3-kube-api-access-jjqql\" (UniqueName: \"kubernetes.io/projected/5257abaf-67a0-45f6-80d2-077bc7e506d3-kube-api-access-jjqql\") pod \"5257abaf-67a0-45f6-80d2-077bc7e506d3\" (UID: \"5257abaf-67a0-45f6-80d2-077bc7e506d3\") " Oct 31 03:52:03.581757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:03.581732 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-openshift-ca" pod "5257abaf-67a0-45f6-80d2-077bc7e506d3" (UID: "5257abaf-67a0-45f6-80d2-077bc7e506d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:03.582057 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:03.582035 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:03.583678 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:03.583648 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-xks-ca" pod "5257abaf-67a0-45f6-80d2-077bc7e506d3" (UID: "5257abaf-67a0-45f6-80d2-077bc7e506d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:52:03.583878 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:03.583855 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5257abaf-67a0-45f6-80d2-077bc7e506d3-kube-api-access-jjqql" pod "5257abaf-67a0-45f6-80d2-077bc7e506d3" (UID: "5257abaf-67a0-45f6-80d2-077bc7e506d3"). InnerVolumeSpecName "kube-api-access-jjqql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:03.683262 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:03.683225 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5257abaf-67a0-45f6-80d2-077bc7e506d3-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:03.683262 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:03.683257 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjqql\" (UniqueName: \"kubernetes.io/projected/5257abaf-67a0-45f6-80d2-077bc7e506d3-kube-api-access-jjqql\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:04.053316 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:04.053280 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h"] Oct 31 03:52:04.060196 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:04.060158 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5709b7d5-5qf2h"] Oct 31 03:52:04.326867 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:04.326783 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" event={"ID":"5257abaf-67a0-45f6-80d2-077bc7e506d3","Type":"ContainerDied","Data":"32aacc047e9f5fd928827a2bec7b86ae32c75a0173843e29d19f32b14e90f6e3"} Oct 31 03:52:04.326867 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:04.326828 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc" Oct 31 03:52:04.327043 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:04.326842 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32aacc047e9f5fd928827a2bec7b86ae32c75a0173843e29d19f32b14e90f6e3" Oct 31 03:52:04.373187 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:52:04.373154 4426 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5257abaf_67a0_45f6_80d2_077bc7e506d3.slice\": RecentStats: unable to find data in memory cache]" Oct 31 03:52:05.131325 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:05.131293 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="773296c4-147e-48e7-876b-f07250adb3f0" path="/var/lib/kubelet/pods/773296c4-147e-48e7-876b-f07250adb3f0/volumes" Oct 31 03:52:06.874956 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:06.873439 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:52:06.894462 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:06.894425 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:52:07.577556 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:07.577523 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:52:08.188522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:08.188472 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm"] Oct 31 03:52:08.189175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:08.189118 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6a8ee93-8hhmm"] Oct 31 03:52:09.132977 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:09.132937 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a6fdbf1-87a8-42f9-8877-f4a5032bd585" path="/var/lib/kubelet/pods/7a6fdbf1-87a8-42f9-8877-f4a5032bd585/volumes" Oct 31 03:52:15.666544 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.666506 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2"] Oct 31 03:52:15.667187 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.667170 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5257abaf-67a0-45f6-80d2-077bc7e506d3" containerName="revoke-keys" Oct 31 03:52:15.667187 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.667187 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5257abaf-67a0-45f6-80d2-077bc7e506d3" containerName="revoke-keys" Oct 31 03:52:15.667325 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.667287 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="5257abaf-67a0-45f6-80d2-077bc7e506d3" containerName="revoke-keys" Oct 31 03:52:15.670463 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.670434 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:15.719922 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.719858 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:52:15.766395 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.766248 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:52:15.766395 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.766275 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:52:15.766395 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.766340 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:52:15.771725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.771697 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2"] Oct 31 03:52:15.809062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.809028 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d7b3a17-rcwb2\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:15.809242 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.809204 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9qw5\" (UniqueName: \"kubernetes.io/projected/711e32e7-8af5-4e8d-83f2-221c67c3b55f-kube-api-access-q9qw5\") pod \"maas-api-revoke-sub-8d7b3a17-rcwb2\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:15.809310 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.809267 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d7b3a17-rcwb2\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:15.910850 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.910809 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d7b3a17-rcwb2\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:15.911041 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.910937 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d7b3a17-rcwb2\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:15.911041 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.911028 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9qw5\" (UniqueName: \"kubernetes.io/projected/711e32e7-8af5-4e8d-83f2-221c67c3b55f-kube-api-access-q9qw5\") pod \"maas-api-revoke-sub-8d7b3a17-rcwb2\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:15.911668 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.911645 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d7b3a17-rcwb2\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:15.913302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.913279 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d7b3a17-rcwb2\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:15.969019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.968939 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9qw5\" (UniqueName: \"kubernetes.io/projected/711e32e7-8af5-4e8d-83f2-221c67c3b55f-kube-api-access-q9qw5\") pod \"maas-api-revoke-sub-8d7b3a17-rcwb2\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:15.984600 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:15.984568 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:16.358577 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:52:16.358537 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711e32e7_8af5_4e8d_83f2_221c67c3b55f.slice/crio-6483736aa18fc4dad1ca6c67115e5c43a76d113e7e2b73e2286c2cdcbfe70427 WatchSource:0}: Error finding container 6483736aa18fc4dad1ca6c67115e5c43a76d113e7e2b73e2286c2cdcbfe70427: Status 404 returned error can't find the container with id 6483736aa18fc4dad1ca6c67115e5c43a76d113e7e2b73e2286c2cdcbfe70427 Oct 31 03:52:16.359664 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:16.359637 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2"] Oct 31 03:52:16.478417 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:16.478380 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" event={"ID":"711e32e7-8af5-4e8d-83f2-221c67c3b55f","Type":"ContainerStarted","Data":"6483736aa18fc4dad1ca6c67115e5c43a76d113e7e2b73e2286c2cdcbfe70427"} Oct 31 03:52:17.488037 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:17.487939 4426 generic.go:356] "Generic (PLEG): container finished" podID="711e32e7-8af5-4e8d-83f2-221c67c3b55f" containerID="7b01df0b2090391a3e49b9d07f6d836d740bcaca490892a0b32c71f5cef19b3f" exitCode=0 Oct 31 03:52:17.488037 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:17.487972 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" event={"ID":"711e32e7-8af5-4e8d-83f2-221c67c3b55f","Type":"ContainerDied","Data":"7b01df0b2090391a3e49b9d07f6d836d740bcaca490892a0b32c71f5cef19b3f"} Oct 31 03:52:18.671739 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:18.671710 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:18.747264 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:18.747220 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711e32e7-8af5-4e8d-83f2-221c67c3b55f-kube-api-access-q9qw5\" (UniqueName: \"kubernetes.io/projected/711e32e7-8af5-4e8d-83f2-221c67c3b55f-kube-api-access-q9qw5\") pod \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " Oct 31 03:52:18.747481 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:18.747332 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-xks-ca\") pod \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " Oct 31 03:52:18.747481 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:18.747412 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-openshift-ca\") pod \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\" (UID: \"711e32e7-8af5-4e8d-83f2-221c67c3b55f\") " Oct 31 03:52:18.747883 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:18.747845 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-openshift-ca" pod "711e32e7-8af5-4e8d-83f2-221c67c3b55f" (UID: "711e32e7-8af5-4e8d-83f2-221c67c3b55f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:18.748018 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:18.747980 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:18.749626 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:18.749562 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711e32e7-8af5-4e8d-83f2-221c67c3b55f-kube-api-access-q9qw5" pod "711e32e7-8af5-4e8d-83f2-221c67c3b55f" (UID: "711e32e7-8af5-4e8d-83f2-221c67c3b55f"). InnerVolumeSpecName "kube-api-access-q9qw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:18.749983 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:18.749965 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-xks-ca" pod "711e32e7-8af5-4e8d-83f2-221c67c3b55f" (UID: "711e32e7-8af5-4e8d-83f2-221c67c3b55f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:52:18.848902 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:18.848860 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9qw5\" (UniqueName: \"kubernetes.io/projected/711e32e7-8af5-4e8d-83f2-221c67c3b55f-kube-api-access-q9qw5\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:18.848902 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:18.848898 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711e32e7-8af5-4e8d-83f2-221c67c3b55f-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:19.048493 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:19.048385 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss"] Oct 31 03:52:19.053958 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:19.053923 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31a7f25-8pdss"] Oct 31 03:52:19.131968 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:19.131933 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a545899-a391-4432-9587-6ac10f7e55a0" path="/var/lib/kubelet/pods/1a545899-a391-4432-9587-6ac10f7e55a0/volumes" Oct 31 03:52:19.507477 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:19.507450 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" Oct 31 03:52:19.507663 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:19.507485 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2" event={"ID":"711e32e7-8af5-4e8d-83f2-221c67c3b55f","Type":"ContainerDied","Data":"6483736aa18fc4dad1ca6c67115e5c43a76d113e7e2b73e2286c2cdcbfe70427"} Oct 31 03:52:19.507663 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:19.507535 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6483736aa18fc4dad1ca6c67115e5c43a76d113e7e2b73e2286c2cdcbfe70427" Oct 31 03:52:29.922286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.922250 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt"] Oct 31 03:52:29.922870 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.922854 4426 cpu_manager.go:414] "containerMap: removing container" podUID="711e32e7-8af5-4e8d-83f2-221c67c3b55f" containerName="revoke-keys" Oct 31 03:52:29.922916 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.922872 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711e32e7-8af5-4e8d-83f2-221c67c3b55f" containerName="revoke-keys" Oct 31 03:52:29.922993 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.922982 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="711e32e7-8af5-4e8d-83f2-221c67c3b55f" containerName="revoke-keys" Oct 31 03:52:29.927913 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.927892 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:29.983397 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.983365 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt"] Oct 31 03:52:29.986287 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.986250 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb7866b-4998-420e-9afb-29ce3903955c-tls-certs\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:29.986459 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.986342 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-home\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:29.986459 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.986395 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-dshm\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:29.986582 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.986481 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-tmp-dir\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:29.986582 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.986541 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-model-cache\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:29.986745 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:29.986590 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q6wg\" (UniqueName: \"kubernetes.io/projected/dcb7866b-4998-420e-9afb-29ce3903955c-kube-api-access-2q6wg\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.022899 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.022868 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a64fd2\"/\"distinct-2-w2-a64fd2-kserve-self-signed-certs\"" Oct 31 03:52:30.090207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.088382 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-tmp-dir\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.090207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.088479 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-model-cache\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.090207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.088554 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q6wg\" (UniqueName: \"kubernetes.io/projected/dcb7866b-4998-420e-9afb-29ce3903955c-kube-api-access-2q6wg\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.090207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.088757 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb7866b-4998-420e-9afb-29ce3903955c-tls-certs\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.090207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.088890 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-home\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.090207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.088973 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-dshm\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.090870 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.090805 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-tmp-dir\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.091161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.091086 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-model-cache\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.092404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.092375 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-home\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.101228 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.099001 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb7866b-4998-420e-9afb-29ce3903955c-tls-certs\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.104237 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.104188 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-dshm\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.147586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.147551 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q6wg\" (UniqueName: \"kubernetes.io/projected/dcb7866b-4998-420e-9afb-29ce3903955c-kube-api-access-2q6wg\") pod \"distinct-2-w2-a64fd2-kserve-699c8d999-64dpt\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.244032 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.243939 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:30.631260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:30.631229 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt"] Oct 31 03:52:30.632817 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:52:30.632786 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcb7866b_4998_420e_9afb_29ce3903955c.slice/crio-9abc14ffc95bbf1468f88ca7772664412786effd03463dcd6d85961624aaef8a WatchSource:0}: Error finding container 9abc14ffc95bbf1468f88ca7772664412786effd03463dcd6d85961624aaef8a: Status 404 returned error can't find the container with id 9abc14ffc95bbf1468f88ca7772664412786effd03463dcd6d85961624aaef8a Oct 31 03:52:31.608190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:31.608146 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" event={"ID":"dcb7866b-4998-420e-9afb-29ce3903955c","Type":"ContainerStarted","Data":"f33cf7346e68bfe4c8345ed244f29f129f07bc13ffa712be663300fd90b75f77"} Oct 31 03:52:31.608613 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:31.608205 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" event={"ID":"dcb7866b-4998-420e-9afb-29ce3903955c","Type":"ContainerStarted","Data":"9abc14ffc95bbf1468f88ca7772664412786effd03463dcd6d85961624aaef8a"} Oct 31 03:52:31.719969 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:31.719914 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podStartSLOduration=2.7199001430000003 podStartE2EDuration="2.719900143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:31.717747796 +0000 UTC m=+1448.718373027" watchObservedRunningTime="2026-10-31 03:52:31.719900143 +0000 UTC m=+1448.720525370" Oct 31 03:52:32.088234 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:32.088192 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:52:32.244306 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:32.244272 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:32.824485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:32.824450 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm"] Oct 31 03:52:32.830314 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:32.830281 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:32.853293 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:52:32.853251 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:52:32.910150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:32.909629 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm"] Oct 31 03:52:32.961601 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:32.959844 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:52:32.961601 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:32.959919 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:52:32.961601 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:32.959981 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:52:32.961601 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:32.960069 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:52:32.965871 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:32.963191 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ccn\" (UniqueName: \"kubernetes.io/projected/f6eceb4e-b68e-4893-8879-60fedaf397b9-kube-api-access-88ccn\") pod \"maas-api-revoke-sub-5fc1c4c3-thjgm\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:33.011281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:33.011231 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc1c4c3-thjgm\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:33.011472 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:33.011424 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc1c4c3-thjgm\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:33.112482 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:33.112386 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ccn\" (UniqueName: \"kubernetes.io/projected/f6eceb4e-b68e-4893-8879-60fedaf397b9-kube-api-access-88ccn\") pod \"maas-api-revoke-sub-5fc1c4c3-thjgm\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:33.112482 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:33.112465 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc1c4c3-thjgm\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:33.112717 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:33.112534 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc1c4c3-thjgm\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:33.113819 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:33.113784 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc1c4c3-thjgm\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:33.115195 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:33.115169 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc1c4c3-thjgm\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:33.172145 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:33.170872 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ccn\" (UniqueName: \"kubernetes.io/projected/f6eceb4e-b68e-4893-8879-60fedaf397b9-kube-api-access-88ccn\") pod \"maas-api-revoke-sub-5fc1c4c3-thjgm\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:33.451736 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:33.450400 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:34.231864 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:34.231817 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm"] Oct 31 03:52:34.894185 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:34.894146 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" event={"ID":"f6eceb4e-b68e-4893-8879-60fedaf397b9","Type":"ContainerStarted","Data":"0c37690a5d0f05bbc90c1a7a36a0ee4203ed6221a5039912e5965490fdb4f264"} Oct 31 03:52:35.118148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:35.116517 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:35.158159 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:35.148263 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:35.162147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:35.158606 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 31 03:52:35.182583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:35.182217 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:35.904411 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:35.904367 4426 generic.go:356] "Generic (PLEG): container finished" podID="f6eceb4e-b68e-4893-8879-60fedaf397b9" containerID="0f2f687120e185e2fc30c4a2bb0ce490493f171af078df9a3291a628642dfc88" exitCode=0 Oct 31 03:52:35.904799 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:35.904441 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" event={"ID":"f6eceb4e-b68e-4893-8879-60fedaf397b9","Type":"ContainerDied","Data":"0f2f687120e185e2fc30c4a2bb0ce490493f171af078df9a3291a628642dfc88"} Oct 31 03:52:37.051727 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.051678 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km"] Oct 31 03:52:37.059998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.059965 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74547e64-zp7km"] Oct 31 03:52:37.098052 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.098026 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:37.132371 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.132339 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d912c1-83b4-448f-bf05-53fd3496625b" path="/var/lib/kubelet/pods/63d912c1-83b4-448f-bf05-53fd3496625b/volumes" Oct 31 03:52:37.137894 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.137866 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-xks-ca\") pod \"f6eceb4e-b68e-4893-8879-60fedaf397b9\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " Oct 31 03:52:37.138056 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.137973 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-openshift-ca\") pod \"f6eceb4e-b68e-4893-8879-60fedaf397b9\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " Oct 31 03:52:37.138160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.138103 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6eceb4e-b68e-4893-8879-60fedaf397b9-kube-api-access-88ccn\" (UniqueName: \"kubernetes.io/projected/f6eceb4e-b68e-4893-8879-60fedaf397b9-kube-api-access-88ccn\") pod \"f6eceb4e-b68e-4893-8879-60fedaf397b9\" (UID: \"f6eceb4e-b68e-4893-8879-60fedaf397b9\") " Oct 31 03:52:37.138399 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.138376 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-openshift-ca" pod "f6eceb4e-b68e-4893-8879-60fedaf397b9" (UID: "f6eceb4e-b68e-4893-8879-60fedaf397b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:37.138690 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.138672 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:37.140004 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.139978 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-xks-ca" pod "f6eceb4e-b68e-4893-8879-60fedaf397b9" (UID: "f6eceb4e-b68e-4893-8879-60fedaf397b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:52:37.140120 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.140095 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6eceb4e-b68e-4893-8879-60fedaf397b9-kube-api-access-88ccn" pod "f6eceb4e-b68e-4893-8879-60fedaf397b9" (UID: "f6eceb4e-b68e-4893-8879-60fedaf397b9"). InnerVolumeSpecName "kube-api-access-88ccn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:37.240229 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.240119 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88ccn\" (UniqueName: \"kubernetes.io/projected/f6eceb4e-b68e-4893-8879-60fedaf397b9-kube-api-access-88ccn\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:37.240229 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.240168 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6eceb4e-b68e-4893-8879-60fedaf397b9-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:37.923838 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.923796 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" event={"ID":"f6eceb4e-b68e-4893-8879-60fedaf397b9","Type":"ContainerDied","Data":"0c37690a5d0f05bbc90c1a7a36a0ee4203ed6221a5039912e5965490fdb4f264"} Oct 31 03:52:37.923990 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.923865 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c37690a5d0f05bbc90c1a7a36a0ee4203ed6221a5039912e5965490fdb4f264" Oct 31 03:52:37.923990 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:37.923812 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm" Oct 31 03:52:40.244260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.244223 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:40.257006 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.256975 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:40.550190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.550102 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf"] Oct 31 03:52:40.550720 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.550706 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f6eceb4e-b68e-4893-8879-60fedaf397b9" containerName="revoke-keys" Oct 31 03:52:40.550768 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.550722 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6eceb4e-b68e-4893-8879-60fedaf397b9" containerName="revoke-keys" Oct 31 03:52:40.550839 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.550830 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6eceb4e-b68e-4893-8879-60fedaf397b9" containerName="revoke-keys" Oct 31 03:52:40.554218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.554199 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:40.579870 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.579836 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91678f80-t5bwf\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:40.580047 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.580006 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91678f80-t5bwf\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:40.580114 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.580064 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd4hp\" (UniqueName: \"kubernetes.io/projected/1592df3f-7096-4d96-85af-0ba30ff950b2-kube-api-access-pd4hp\") pod \"maas-api-revoke-sub-91678f80-t5bwf\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:40.603857 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.603823 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:52:40.603857 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.603850 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:52:40.603857 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.603823 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:52:40.604155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.603922 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:52:40.611111 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.611085 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf"] Oct 31 03:52:40.681059 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.681020 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4hp\" (UniqueName: \"kubernetes.io/projected/1592df3f-7096-4d96-85af-0ba30ff950b2-kube-api-access-pd4hp\") pod \"maas-api-revoke-sub-91678f80-t5bwf\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:40.681255 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.681154 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91678f80-t5bwf\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:40.681255 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.681246 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91678f80-t5bwf\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:40.681839 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.681808 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91678f80-t5bwf\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:40.683742 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.683714 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91678f80-t5bwf\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:40.738771 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.738735 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd4hp\" (UniqueName: \"kubernetes.io/projected/1592df3f-7096-4d96-85af-0ba30ff950b2-kube-api-access-pd4hp\") pod \"maas-api-revoke-sub-91678f80-t5bwf\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:40.868924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.868836 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:40.991390 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:40.991119 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:52:41.270406 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:41.270369 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf"] Oct 31 03:52:41.273815 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:52:41.273781 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1592df3f_7096_4d96_85af_0ba30ff950b2.slice/crio-8a2d64ddb0c08adbd823b3b5cbb55b07a39495def22f7733cdf39bc9afc8ba96 WatchSource:0}: Error finding container 8a2d64ddb0c08adbd823b3b5cbb55b07a39495def22f7733cdf39bc9afc8ba96: Status 404 returned error can't find the container with id 8a2d64ddb0c08adbd823b3b5cbb55b07a39495def22f7733cdf39bc9afc8ba96 Oct 31 03:52:41.988758 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:41.988717 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" event={"ID":"1592df3f-7096-4d96-85af-0ba30ff950b2","Type":"ContainerStarted","Data":"35f6ec230b75c4eb30edfd0eed85c4ebf235a29c5ae63a44a413c2bfdcbdd2e7"} Oct 31 03:52:41.988960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:41.988779 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" event={"ID":"1592df3f-7096-4d96-85af-0ba30ff950b2","Type":"ContainerStarted","Data":"8a2d64ddb0c08adbd823b3b5cbb55b07a39495def22f7733cdf39bc9afc8ba96"} Oct 31 03:52:42.100812 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:42.100758 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" podStartSLOduration=2.10073497 podStartE2EDuration="2.10073497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:42.0973669 +0000 UTC m=+1459.097992130" watchObservedRunningTime="2026-10-31 03:52:42.10073497 +0000 UTC m=+1459.101360197" Oct 31 03:52:43.001951 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:43.001918 4426 generic.go:356] "Generic (PLEG): container finished" podID="1592df3f-7096-4d96-85af-0ba30ff950b2" containerID="35f6ec230b75c4eb30edfd0eed85c4ebf235a29c5ae63a44a413c2bfdcbdd2e7" exitCode=0 Oct 31 03:52:43.002475 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:43.002007 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" event={"ID":"1592df3f-7096-4d96-85af-0ba30ff950b2","Type":"ContainerDied","Data":"35f6ec230b75c4eb30edfd0eed85c4ebf235a29c5ae63a44a413c2bfdcbdd2e7"} Oct 31 03:52:44.181764 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:44.181732 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:44.225471 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:44.225432 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-xks-ca\") pod \"1592df3f-7096-4d96-85af-0ba30ff950b2\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " Oct 31 03:52:44.225655 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:44.225489 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1592df3f-7096-4d96-85af-0ba30ff950b2-kube-api-access-pd4hp\" (UniqueName: \"kubernetes.io/projected/1592df3f-7096-4d96-85af-0ba30ff950b2-kube-api-access-pd4hp\") pod \"1592df3f-7096-4d96-85af-0ba30ff950b2\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " Oct 31 03:52:44.225655 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:44.225570 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-openshift-ca\") pod \"1592df3f-7096-4d96-85af-0ba30ff950b2\" (UID: \"1592df3f-7096-4d96-85af-0ba30ff950b2\") " Oct 31 03:52:44.226026 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:44.225991 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-openshift-ca" pod "1592df3f-7096-4d96-85af-0ba30ff950b2" (UID: "1592df3f-7096-4d96-85af-0ba30ff950b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:44.226602 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:44.226317 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:44.228213 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:44.228190 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1592df3f-7096-4d96-85af-0ba30ff950b2-kube-api-access-pd4hp" pod "1592df3f-7096-4d96-85af-0ba30ff950b2" (UID: "1592df3f-7096-4d96-85af-0ba30ff950b2"). InnerVolumeSpecName "kube-api-access-pd4hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:44.228213 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:44.228190 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-xks-ca" pod "1592df3f-7096-4d96-85af-0ba30ff950b2" (UID: "1592df3f-7096-4d96-85af-0ba30ff950b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:52:44.327560 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:44.327460 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1592df3f-7096-4d96-85af-0ba30ff950b2-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:44.327560 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:44.327493 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd4hp\" (UniqueName: \"kubernetes.io/projected/1592df3f-7096-4d96-85af-0ba30ff950b2-kube-api-access-pd4hp\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:45.021729 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:45.021697 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" Oct 31 03:52:45.021729 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:45.021715 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf" event={"ID":"1592df3f-7096-4d96-85af-0ba30ff950b2","Type":"ContainerDied","Data":"8a2d64ddb0c08adbd823b3b5cbb55b07a39495def22f7733cdf39bc9afc8ba96"} Oct 31 03:52:45.021952 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:45.021766 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a2d64ddb0c08adbd823b3b5cbb55b07a39495def22f7733cdf39bc9afc8ba96" Oct 31 03:52:47.650871 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.650835 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q"] Oct 31 03:52:47.652781 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.652749 4426 cpu_manager.go:414] "containerMap: removing container" podUID="1592df3f-7096-4d96-85af-0ba30ff950b2" containerName="revoke-keys" Oct 31 03:52:47.652948 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.652935 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1592df3f-7096-4d96-85af-0ba30ff950b2" containerName="revoke-keys" Oct 31 03:52:47.653432 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.653412 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="1592df3f-7096-4d96-85af-0ba30ff950b2" containerName="revoke-keys" Oct 31 03:52:47.658938 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.658911 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:47.706471 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.706437 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q"] Oct 31 03:52:47.708169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.708116 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:52:47.708169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.708164 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:52:47.708420 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.708397 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:52:47.746019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.745978 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:52:47.773734 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.773698 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5d30172-qjg9q\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:47.773899 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.773758 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5d30172-qjg9q\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:47.773899 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.773893 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q96cw\" (UniqueName: \"kubernetes.io/projected/3a274751-c231-417f-a8c1-799655f5ef63-kube-api-access-q96cw\") pod \"maas-api-revoke-sub-f5d30172-qjg9q\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:47.874814 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.874770 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q96cw\" (UniqueName: \"kubernetes.io/projected/3a274751-c231-417f-a8c1-799655f5ef63-kube-api-access-q96cw\") pod \"maas-api-revoke-sub-f5d30172-qjg9q\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:47.875010 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.874936 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5d30172-qjg9q\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:47.875010 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.874984 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5d30172-qjg9q\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:47.875643 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.875619 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5d30172-qjg9q\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:47.877367 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.877348 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5d30172-qjg9q\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:47.932860 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.932779 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q96cw\" (UniqueName: \"kubernetes.io/projected/3a274751-c231-417f-a8c1-799655f5ef63-kube-api-access-q96cw\") pod \"maas-api-revoke-sub-f5d30172-qjg9q\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:47.974478 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:47.974446 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:48.366519 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:48.366489 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q"] Oct 31 03:52:48.367588 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:52:48.367561 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a274751_c231_417f_a8c1_799655f5ef63.slice/crio-ca42104debc56c80ff96df31107cb3e723fcdc05af57e499573671c622c239fb WatchSource:0}: Error finding container ca42104debc56c80ff96df31107cb3e723fcdc05af57e499573671c622c239fb: Status 404 returned error can't find the container with id ca42104debc56c80ff96df31107cb3e723fcdc05af57e499573671c622c239fb Oct 31 03:52:49.049194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:49.049155 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr"] Oct 31 03:52:49.068790 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:49.068752 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" event={"ID":"3a274751-c231-417f-a8c1-799655f5ef63","Type":"ContainerStarted","Data":"829c203cf421750f5f6807061851f56eb31ab36691e5713fa5d56a145b7073e5"} Oct 31 03:52:49.068961 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:49.068807 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" event={"ID":"3a274751-c231-417f-a8c1-799655f5ef63","Type":"ContainerStarted","Data":"ca42104debc56c80ff96df31107cb3e723fcdc05af57e499573671c622c239fb"} Oct 31 03:52:49.094233 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:49.094200 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa38e661-p5vqr"] Oct 31 03:52:49.132220 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:49.132185 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53339386-561a-4a09-a76c-480ee83cc23e" path="/var/lib/kubelet/pods/53339386-561a-4a09-a76c-480ee83cc23e/volumes" Oct 31 03:52:49.179162 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:49.179062 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" podStartSLOduration=2.17904801 podStartE2EDuration="2.17904801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:49.177014685 +0000 UTC m=+1466.177639912" watchObservedRunningTime="2026-10-31 03:52:49.17904801 +0000 UTC m=+1466.179673267" Oct 31 03:52:50.078784 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:50.078745 4426 generic.go:356] "Generic (PLEG): container finished" podID="3a274751-c231-417f-a8c1-799655f5ef63" containerID="829c203cf421750f5f6807061851f56eb31ab36691e5713fa5d56a145b7073e5" exitCode=0 Oct 31 03:52:50.079274 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:50.078867 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" event={"ID":"3a274751-c231-417f-a8c1-799655f5ef63","Type":"ContainerDied","Data":"829c203cf421750f5f6807061851f56eb31ab36691e5713fa5d56a145b7073e5"} Oct 31 03:52:51.495973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:51.495937 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:51.629823 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:51.628824 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a274751-c231-417f-a8c1-799655f5ef63-kube-api-access-q96cw\" (UniqueName: \"kubernetes.io/projected/3a274751-c231-417f-a8c1-799655f5ef63-kube-api-access-q96cw\") pod \"3a274751-c231-417f-a8c1-799655f5ef63\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " Oct 31 03:52:51.629823 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:51.628880 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-xks-ca\") pod \"3a274751-c231-417f-a8c1-799655f5ef63\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " Oct 31 03:52:51.629823 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:51.628961 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-openshift-ca\") pod \"3a274751-c231-417f-a8c1-799655f5ef63\" (UID: \"3a274751-c231-417f-a8c1-799655f5ef63\") " Oct 31 03:52:51.658201 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:51.658117 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-xks-ca" pod "3a274751-c231-417f-a8c1-799655f5ef63" (UID: "3a274751-c231-417f-a8c1-799655f5ef63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:52:51.658824 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:51.658781 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a274751-c231-417f-a8c1-799655f5ef63-kube-api-access-q96cw" pod "3a274751-c231-417f-a8c1-799655f5ef63" (UID: "3a274751-c231-417f-a8c1-799655f5ef63"). InnerVolumeSpecName "kube-api-access-q96cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:51.743162 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:51.713381 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-openshift-ca" pod "3a274751-c231-417f-a8c1-799655f5ef63" (UID: "3a274751-c231-417f-a8c1-799655f5ef63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:51.808187 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:51.746833 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q96cw\" (UniqueName: \"kubernetes.io/projected/3a274751-c231-417f-a8c1-799655f5ef63-kube-api-access-q96cw\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:51.808187 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:51.746872 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:51.808187 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:51.746887 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a274751-c231-417f-a8c1-799655f5ef63-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:52.150161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.148423 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" event={"ID":"3a274751-c231-417f-a8c1-799655f5ef63","Type":"ContainerDied","Data":"ca42104debc56c80ff96df31107cb3e723fcdc05af57e499573671c622c239fb"} Oct 31 03:52:52.150161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.148499 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca42104debc56c80ff96df31107cb3e723fcdc05af57e499573671c622c239fb" Oct 31 03:52:52.150161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.148583 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q" Oct 31 03:52:52.644880 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.644827 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52"] Oct 31 03:52:52.658263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.658160 4426 cpu_manager.go:414] "containerMap: removing container" podUID="3a274751-c231-417f-a8c1-799655f5ef63" containerName="revoke-keys" Oct 31 03:52:52.658263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.658190 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a274751-c231-417f-a8c1-799655f5ef63" containerName="revoke-keys" Oct 31 03:52:52.658529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.658383 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a274751-c231-417f-a8c1-799655f5ef63" containerName="revoke-keys" Oct 31 03:52:52.663888 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.663860 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.728220 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.728170 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52"] Oct 31 03:52:52.849404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.849363 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-podinfo\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.849608 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.849416 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-envoy\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.849608 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.849446 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-certs\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.849608 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.849474 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hhg\" (UniqueName: \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-kube-api-access-47hhg\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.849608 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.849518 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.849608 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.849575 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-credential-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.849874 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.849635 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-token\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.849874 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.849725 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-data\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.849874 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.849827 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istiod-ca-cert\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951241 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951143 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hhg\" (UniqueName: \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-kube-api-access-47hhg\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951241 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951215 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951484 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951260 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-credential-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951484 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951303 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-token\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951484 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951385 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-data\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951496 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istiod-ca-cert\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951592 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-podinfo\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951626 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-envoy\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951806 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951655 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-certs\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951806 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951726 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951806 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951764 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-credential-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.951965 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951797 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-data\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.952018 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.951965 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-certs\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.952329 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.952307 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istiod-ca-cert\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.954000 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.953978 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-envoy\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:52.954098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:52.954081 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-podinfo\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:53.010548 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.010513 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t"] Oct 31 03:52:53.011975 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.011946 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hhg\" (UniqueName: \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-kube-api-access-47hhg\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:53.012247 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.012222 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-token\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:53.016873 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.016848 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.044719 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.044693 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h"] Oct 31 03:52:53.049305 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.049282 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.077552 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.077518 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t"] Oct 31 03:52:53.120513 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.118482 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h"] Oct 31 03:52:53.156649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.156587 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-certs\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.156851 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.156665 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-envoy\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.156851 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.156718 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44dd1297-446c-439e-99aa-87097742d386-istio-podinfo\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.156851 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.156761 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.156851 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.156794 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-credential-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.156851 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.156831 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44dd1297-446c-439e-99aa-87097742d386-istiod-ca-cert\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.157161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.156870 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-data\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.157161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.156926 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-data\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.157161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.156963 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-envoy\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.157161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.157029 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-podinfo\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.157161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.157060 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-credential-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.157161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.157148 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-certs\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.157495 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.157184 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2f7796-6471-497b-a49a-5c13e85e4213-istiod-ca-cert\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.157495 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.157254 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.157495 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.157287 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9svb8\" (UniqueName: \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-kube-api-access-9svb8\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.157495 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.157413 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-istio-token\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.157495 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.157481 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-token\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.157756 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.157531 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgrrc\" (UniqueName: \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-kube-api-access-hgrrc\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.258924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.258821 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-istio-token\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.258924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.258864 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-token\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.258924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.258885 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgrrc\" (UniqueName: \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-kube-api-access-hgrrc\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.259260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.258942 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-certs\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.259260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.258960 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-envoy\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.259260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.258988 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44dd1297-446c-439e-99aa-87097742d386-istio-podinfo\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.259260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259032 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.259260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259057 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-credential-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.259260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259086 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44dd1297-446c-439e-99aa-87097742d386-istiod-ca-cert\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.259260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259117 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-data\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.259260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259185 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-data\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.259260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259210 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-envoy\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.259260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259269 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-podinfo\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.259791 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259293 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-credential-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.259791 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259348 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-certs\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.259791 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259383 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2f7796-6471-497b-a49a-5c13e85e4213-istiod-ca-cert\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.259791 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259440 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.259791 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259466 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9svb8\" (UniqueName: \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-kube-api-access-9svb8\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.259791 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259556 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-credential-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.260116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259833 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-credential-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.260116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.259900 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-certs\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.260116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.260050 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.260299 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.260111 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-data\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.260354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.260295 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-data\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.260549 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.260527 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-certs\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.260681 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.260645 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-socket\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.260681 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.260586 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44dd1297-446c-439e-99aa-87097742d386-istiod-ca-cert\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.261202 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.261177 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2f7796-6471-497b-a49a-5c13e85e4213-istiod-ca-cert\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.261751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.261729 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-envoy\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.262236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.262208 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44dd1297-446c-439e-99aa-87097742d386-istio-podinfo\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.262345 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.262296 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-envoy\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.262862 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.262839 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-podinfo\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.297489 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.297455 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:53.316934 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.316837 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-token\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.317495 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.317466 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-istio-token\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.317707 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.317681 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgrrc\" (UniqueName: \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-kube-api-access-hgrrc\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.318410 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.318384 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9svb8\" (UniqueName: \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-kube-api-access-9svb8\") pod \"e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.348788 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.348748 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:53.364453 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.364423 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:53.738853 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.738820 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52"] Oct 31 03:52:53.779981 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:52:53.779949 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5462e40_4af4_4dc9_9a84_4169c3e9fd3b.slice/crio-b51bd8a768606007e8dae62973479437d2b81a65fd8b14fc2e74478c8ccb9f8e WatchSource:0}: Error finding container b51bd8a768606007e8dae62973479437d2b81a65fd8b14fc2e74478c8ccb9f8e: Status 404 returned error can't find the container with id b51bd8a768606007e8dae62973479437d2b81a65fd8b14fc2e74478c8ccb9f8e Oct 31 03:52:53.782667 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.782623 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:52:53.782788 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.782717 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:52:53.782788 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.782757 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:52:53.794097 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.794071 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h"] Oct 31 03:52:53.797446 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:52:53.797423 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dd1297_446c_439e_99aa_87097742d386.slice/crio-fc80523a0e243340fcf31a52b224b1dda00c34126ab10910e6408040be6f86c9 WatchSource:0}: Error finding container fc80523a0e243340fcf31a52b224b1dda00c34126ab10910e6408040be6f86c9: Status 404 returned error can't find the container with id fc80523a0e243340fcf31a52b224b1dda00c34126ab10910e6408040be6f86c9 Oct 31 03:52:53.800408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.800377 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:52:53.800507 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.800448 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:52:53.800507 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.800497 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:52:53.842064 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.842035 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t"] Oct 31 03:52:53.846276 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:52:53.846241 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2f7796_6471_497b_a49a_5c13e85e4213.slice/crio-02040eeb92e14de37985f9ac642e6b0bbd7a4acd21ec3fd97678970d7b51dd5b WatchSource:0}: Error finding container 02040eeb92e14de37985f9ac642e6b0bbd7a4acd21ec3fd97678970d7b51dd5b: Status 404 returned error can't find the container with id 02040eeb92e14de37985f9ac642e6b0bbd7a4acd21ec3fd97678970d7b51dd5b Oct 31 03:52:53.849676 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.849641 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:52:53.849770 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.849711 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:52:53.849770 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:53.849740 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:52:54.183098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.183057 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" event={"ID":"44dd1297-446c-439e-99aa-87097742d386","Type":"ContainerStarted","Data":"6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787"} Oct 31 03:52:54.183328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.183162 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" event={"ID":"44dd1297-446c-439e-99aa-87097742d386","Type":"ContainerStarted","Data":"fc80523a0e243340fcf31a52b224b1dda00c34126ab10910e6408040be6f86c9"} Oct 31 03:52:54.186644 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.186590 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" event={"ID":"7c2f7796-6471-497b-a49a-5c13e85e4213","Type":"ContainerStarted","Data":"8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f"} Oct 31 03:52:54.186868 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.186847 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" event={"ID":"7c2f7796-6471-497b-a49a-5c13e85e4213","Type":"ContainerStarted","Data":"02040eeb92e14de37985f9ac642e6b0bbd7a4acd21ec3fd97678970d7b51dd5b"} Oct 31 03:52:54.189062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.189031 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" event={"ID":"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b","Type":"ContainerStarted","Data":"7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353"} Oct 31 03:52:54.189211 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.189089 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" event={"ID":"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b","Type":"ContainerStarted","Data":"b51bd8a768606007e8dae62973479437d2b81a65fd8b14fc2e74478c8ccb9f8e"} Oct 31 03:52:54.298473 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.298435 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:52:54.300044 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.300011 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:52:54.300228 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.300073 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:52:54.300447 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.300403 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" podStartSLOduration=2.300388482 podStartE2EDuration="2.300388482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:54.296466247 +0000 UTC m=+1471.297091478" watchObservedRunningTime="2026-10-31 03:52:54.300388482 +0000 UTC m=+1471.301013711" Oct 31 03:52:54.349768 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.349734 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:52:54.352166 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.352120 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:52:54.352335 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.352219 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:52:54.365953 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.365630 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:52:54.367790 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.367759 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:52:54.368003 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.367981 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:52:54.411607 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.411555 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" podStartSLOduration=2.411539665 podStartE2EDuration="2.411539665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:54.408930802 +0000 UTC m=+1471.409556033" watchObservedRunningTime="2026-10-31 03:52:54.411539665 +0000 UTC m=+1471.412164893" Oct 31 03:52:54.577205 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:54.571870 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podStartSLOduration=2.571845788 podStartE2EDuration="2.571845788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:54.534899639 +0000 UTC m=+1471.535524868" watchObservedRunningTime="2026-10-31 03:52:54.571845788 +0000 UTC m=+1471.572471017" Oct 31 03:52:55.307153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:55.302758 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:52:55.307153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:55.302856 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:52:55.350197 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:55.350158 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:52:55.350497 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:55.350460 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:52:55.383627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:55.383558 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:52:55.383846 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:55.383633 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:52:56.301278 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:56.300779 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:52:56.301278 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:56.300839 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:52:56.366151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:56.351514 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:52:56.366151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:56.351577 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:52:56.366751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:56.366589 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:52:56.366751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:56.366667 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:52:57.298413 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.298374 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:52:57.298708 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.298681 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:52:57.351149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.351012 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:52:57.351149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.351083 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:52:57.365688 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.365554 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:52:57.365688 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.365625 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:52:57.575367 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.575239 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz"] Oct 31 03:52:57.598560 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.598533 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:52:57.618150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.614898 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz"] Oct 31 03:52:57.662428 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.655480 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:52:57.662428 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.655543 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:52:57.662428 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.655595 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:52:57.662428 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.655684 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:52:57.798873 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.798826 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e98a52-wc2zz\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:52:57.799251 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.799223 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69mj\" (UniqueName: \"kubernetes.io/projected/97cf9175-4a05-412c-b2f5-324e2c227639-kube-api-access-j69mj\") pod \"maas-api-revoke-sub-a5e98a52-wc2zz\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:52:57.799496 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.799479 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e98a52-wc2zz\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:52:57.901425 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.901385 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69mj\" (UniqueName: \"kubernetes.io/projected/97cf9175-4a05-412c-b2f5-324e2c227639-kube-api-access-j69mj\") pod \"maas-api-revoke-sub-a5e98a52-wc2zz\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:52:57.902144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.902105 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e98a52-wc2zz\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:52:57.902366 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.902346 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e98a52-wc2zz\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:52:57.902681 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.902656 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e98a52-wc2zz\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:52:57.904812 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.904785 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e98a52-wc2zz\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:52:57.978149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:57.976889 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69mj\" (UniqueName: \"kubernetes.io/projected/97cf9175-4a05-412c-b2f5-324e2c227639-kube-api-access-j69mj\") pod \"maas-api-revoke-sub-a5e98a52-wc2zz\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:52:58.091155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:58.090181 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w"] Oct 31 03:52:58.097152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:58.096911 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704dcc4a-cpx8w"] Oct 31 03:52:58.255233 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:58.254346 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:52:58.300293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:58.300259 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:52:58.300469 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:58.300305 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:52:58.371151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:58.362459 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:52:58.371151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:58.362519 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:52:58.411176 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:58.399856 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:52:58.411176 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:58.399930 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:52:59.306168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:59.302258 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:52:59.306168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:59.302328 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:52:59.379679 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:59.379639 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:52:59.545215 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:59.522384 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:52:59.547331 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:59.439659 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb1caa6-9b1d-45de-912d-83b7af928e4d" path="/var/lib/kubelet/pods/aeb1caa6-9b1d-45de-912d-83b7af928e4d/volumes" Oct 31 03:52:59.573169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:59.547918 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz"] Oct 31 03:52:59.573169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:59.390248 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:52:59.573169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:59.548036 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:52:59.804893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:52:59.804698 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" event={"ID":"97cf9175-4a05-412c-b2f5-324e2c227639","Type":"ContainerStarted","Data":"b74497e633ef72795393b127e17904b7dfc9758db86c2a99b196eb0d9102f71d"} Oct 31 03:53:00.182151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:00.181188 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n"] Oct 31 03:53:00.230585 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:00.230215 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383900a6-vz24n"] Oct 31 03:53:00.306165 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:00.299519 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:53:00.306165 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:00.299589 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:53:00.349389 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:00.349210 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:53:00.349389 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:00.349278 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:53:00.365079 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:00.365042 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:53:00.365303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:00.365104 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:53:00.816193 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:00.816142 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" event={"ID":"97cf9175-4a05-412c-b2f5-324e2c227639","Type":"ContainerStarted","Data":"31f8d35b1150d601b844fb73481e4718e6074e109dd3097e21cb1ffdfe3cfa34"} Oct 31 03:53:00.937158 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:00.933423 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" podStartSLOduration=3.93340192 podStartE2EDuration="3.93340192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:00.924683896 +0000 UTC m=+1477.925309124" watchObservedRunningTime="2026-10-31 03:53:00.93340192 +0000 UTC m=+1477.934027149" Oct 31 03:53:01.135203 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:01.134587 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5" path="/var/lib/kubelet/pods/8e9f6e0f-dfd5-48a5-8b23-c921247f0bd5/volumes" Oct 31 03:53:01.298193 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:01.298149 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:53:01.298498 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:01.298465 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:53:01.353154 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:01.351591 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:53:01.353154 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:01.351653 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:53:01.834152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:01.833810 4426 generic.go:356] "Generic (PLEG): container finished" podID="97cf9175-4a05-412c-b2f5-324e2c227639" containerID="31f8d35b1150d601b844fb73481e4718e6074e109dd3097e21cb1ffdfe3cfa34" exitCode=0 Oct 31 03:53:01.834366 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:01.834185 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" event={"ID":"97cf9175-4a05-412c-b2f5-324e2c227639","Type":"ContainerDied","Data":"31f8d35b1150d601b844fb73481e4718e6074e109dd3097e21cb1ffdfe3cfa34"} Oct 31 03:53:02.036223 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:02.036181 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:53:02.038006 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:02.037982 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:53:02.038804 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:02.038784 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 03:53:02.303340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:02.303307 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:53:02.303807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:02.303782 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:53:02.305675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:02.304782 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 03:53:02.354274 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:02.354244 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:53:02.354725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:02.354553 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:53:02.355377 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:02.355361 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 03:53:02.853950 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:53:02.853911 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:53:03.036078 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.036047 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:53:03.151240 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.151209 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-openshift-ca\") pod \"97cf9175-4a05-412c-b2f5-324e2c227639\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " Oct 31 03:53:03.151442 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.151250 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97cf9175-4a05-412c-b2f5-324e2c227639-kube-api-access-j69mj\" (UniqueName: \"kubernetes.io/projected/97cf9175-4a05-412c-b2f5-324e2c227639-kube-api-access-j69mj\") pod \"97cf9175-4a05-412c-b2f5-324e2c227639\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " Oct 31 03:53:03.151509 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.151492 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-xks-ca\") pod \"97cf9175-4a05-412c-b2f5-324e2c227639\" (UID: \"97cf9175-4a05-412c-b2f5-324e2c227639\") " Oct 31 03:53:03.151979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.151946 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-openshift-ca" pod "97cf9175-4a05-412c-b2f5-324e2c227639" (UID: "97cf9175-4a05-412c-b2f5-324e2c227639"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:03.153965 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.153939 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cf9175-4a05-412c-b2f5-324e2c227639-kube-api-access-j69mj" pod "97cf9175-4a05-412c-b2f5-324e2c227639" (UID: "97cf9175-4a05-412c-b2f5-324e2c227639"). InnerVolumeSpecName "kube-api-access-j69mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:03.154056 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.153940 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-xks-ca" pod "97cf9175-4a05-412c-b2f5-324e2c227639" (UID: "97cf9175-4a05-412c-b2f5-324e2c227639"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:03.252879 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.252841 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:03.252879 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.252876 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97cf9175-4a05-412c-b2f5-324e2c227639-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:03.252879 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.252886 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j69mj\" (UniqueName: \"kubernetes.io/projected/97cf9175-4a05-412c-b2f5-324e2c227639-kube-api-access-j69mj\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:03.857689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.857657 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" Oct 31 03:53:03.857689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.857678 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz" event={"ID":"97cf9175-4a05-412c-b2f5-324e2c227639","Type":"ContainerDied","Data":"b74497e633ef72795393b127e17904b7dfc9758db86c2a99b196eb0d9102f71d"} Oct 31 03:53:03.858247 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:03.857739 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b74497e633ef72795393b127e17904b7dfc9758db86c2a99b196eb0d9102f71d" Oct 31 03:53:06.101234 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.101144 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5"] Oct 31 03:53:06.101812 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.101798 4426 cpu_manager.go:414] "containerMap: removing container" podUID="97cf9175-4a05-412c-b2f5-324e2c227639" containerName="revoke-keys" Oct 31 03:53:06.101852 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.101813 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cf9175-4a05-412c-b2f5-324e2c227639" containerName="revoke-keys" Oct 31 03:53:06.101914 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.101905 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cf9175-4a05-412c-b2f5-324e2c227639" containerName="revoke-keys" Oct 31 03:53:06.105853 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.105833 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:06.155365 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.155272 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:53:06.193518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.193462 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34587b81-6nhv5\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:06.193721 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.193610 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54s6f\" (UniqueName: \"kubernetes.io/projected/87beeef2-c988-4981-8f94-955df7f7879b-kube-api-access-54s6f\") pod \"maas-api-revoke-sub-34587b81-6nhv5\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:06.193796 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.193773 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34587b81-6nhv5\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:06.201649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.201608 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:53:06.201649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.201640 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:53:06.201945 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.201925 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:53:06.207118 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.207093 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5"] Oct 31 03:53:06.295340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.295293 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34587b81-6nhv5\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:06.295552 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.295379 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54s6f\" (UniqueName: \"kubernetes.io/projected/87beeef2-c988-4981-8f94-955df7f7879b-kube-api-access-54s6f\") pod \"maas-api-revoke-sub-34587b81-6nhv5\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:06.295552 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.295466 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34587b81-6nhv5\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:06.296267 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.296240 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34587b81-6nhv5\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:06.297929 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.297899 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34587b81-6nhv5\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:06.353757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.353673 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54s6f\" (UniqueName: \"kubernetes.io/projected/87beeef2-c988-4981-8f94-955df7f7879b-kube-api-access-54s6f\") pod \"maas-api-revoke-sub-34587b81-6nhv5\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:06.421142 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.421087 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:06.818735 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.818699 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5"] Oct 31 03:53:06.822815 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:06.822774 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87beeef2_c988_4981_8f94_955df7f7879b.slice/crio-ff46cd34e497ca6b10c1e2067ef17fef2ea4a55930a886908b662094da1d3654 WatchSource:0}: Error finding container ff46cd34e497ca6b10c1e2067ef17fef2ea4a55930a886908b662094da1d3654: Status 404 returned error can't find the container with id ff46cd34e497ca6b10c1e2067ef17fef2ea4a55930a886908b662094da1d3654 Oct 31 03:53:06.889056 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:06.889015 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" event={"ID":"87beeef2-c988-4981-8f94-955df7f7879b","Type":"ContainerStarted","Data":"ff46cd34e497ca6b10c1e2067ef17fef2ea4a55930a886908b662094da1d3654"} Oct 31 03:53:07.899113 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:07.899081 4426 generic.go:356] "Generic (PLEG): container finished" podID="87beeef2-c988-4981-8f94-955df7f7879b" containerID="31cb679998ebd7bbb6c34c63fb8fb2077afd2ad063db084df2e2984419ed7e4f" exitCode=0 Oct 31 03:53:07.899564 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:07.899157 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" event={"ID":"87beeef2-c988-4981-8f94-955df7f7879b","Type":"ContainerDied","Data":"31cb679998ebd7bbb6c34c63fb8fb2077afd2ad063db084df2e2984419ed7e4f"} Oct 31 03:53:09.086622 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.086589 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:09.244342 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.244245 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-openshift-ca\") pod \"87beeef2-c988-4981-8f94-955df7f7879b\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " Oct 31 03:53:09.244537 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.244519 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87beeef2-c988-4981-8f94-955df7f7879b-kube-api-access-54s6f\" (UniqueName: \"kubernetes.io/projected/87beeef2-c988-4981-8f94-955df7f7879b-kube-api-access-54s6f\") pod \"87beeef2-c988-4981-8f94-955df7f7879b\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " Oct 31 03:53:09.244617 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.244566 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-xks-ca\") pod \"87beeef2-c988-4981-8f94-955df7f7879b\" (UID: \"87beeef2-c988-4981-8f94-955df7f7879b\") " Oct 31 03:53:09.244673 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.244617 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-openshift-ca" pod "87beeef2-c988-4981-8f94-955df7f7879b" (UID: "87beeef2-c988-4981-8f94-955df7f7879b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:09.245051 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.245029 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:09.246851 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.246821 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-xks-ca" pod "87beeef2-c988-4981-8f94-955df7f7879b" (UID: "87beeef2-c988-4981-8f94-955df7f7879b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:09.247329 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.247300 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87beeef2-c988-4981-8f94-955df7f7879b-kube-api-access-54s6f" pod "87beeef2-c988-4981-8f94-955df7f7879b" (UID: "87beeef2-c988-4981-8f94-955df7f7879b"). InnerVolumeSpecName "kube-api-access-54s6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:09.335551 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.335517 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv"] Oct 31 03:53:09.336606 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.336581 4426 cpu_manager.go:414] "containerMap: removing container" podUID="87beeef2-c988-4981-8f94-955df7f7879b" containerName="revoke-keys" Oct 31 03:53:09.336606 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.336607 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87beeef2-c988-4981-8f94-955df7f7879b" containerName="revoke-keys" Oct 31 03:53:09.336767 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.336758 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="87beeef2-c988-4981-8f94-955df7f7879b" containerName="revoke-keys" Oct 31 03:53:09.341302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.341277 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.346377 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.346347 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54s6f\" (UniqueName: \"kubernetes.io/projected/87beeef2-c988-4981-8f94-955df7f7879b-kube-api-access-54s6f\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:09.346469 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.346382 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87beeef2-c988-4981-8f94-955df7f7879b-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:09.397197 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.397159 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv"] Oct 31 03:53:09.447432 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.447388 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-credential-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.447627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.447494 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.447627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.447529 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-token\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.447765 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.447663 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-certs\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.447765 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.447711 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-envoy\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.447765 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.447738 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-data\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.447928 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.447770 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-podinfo\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.447928 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.447835 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istiod-ca-cert\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.447928 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.447878 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgkwb\" (UniqueName: \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-kube-api-access-mgkwb\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.549730 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.549619 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-credential-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.549730 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.549705 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.549730 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.549728 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-token\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.550019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.549828 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-certs\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.550019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.549865 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-envoy\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.550019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.549891 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-data\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.550019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.549921 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-podinfo\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.550245 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.550085 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istiod-ca-cert\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.550245 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.550151 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-credential-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.550245 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.550218 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgkwb\" (UniqueName: \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-kube-api-access-mgkwb\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.550589 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.550559 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-data\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.550804 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.550779 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-certs\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.550977 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.550953 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.551488 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.551466 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istiod-ca-cert\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.552576 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.552553 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-envoy\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.552897 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.552873 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-podinfo\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.610210 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.610178 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-token\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.610954 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.610929 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgkwb\" (UniqueName: \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-kube-api-access-mgkwb\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.662175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.662120 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:09.917390 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.917356 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" Oct 31 03:53:09.917576 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.917382 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5" event={"ID":"87beeef2-c988-4981-8f94-955df7f7879b","Type":"ContainerDied","Data":"ff46cd34e497ca6b10c1e2067ef17fef2ea4a55930a886908b662094da1d3654"} Oct 31 03:53:09.917576 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:09.917445 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff46cd34e497ca6b10c1e2067ef17fef2ea4a55930a886908b662094da1d3654" Oct 31 03:53:10.104480 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:10.104445 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv"] Oct 31 03:53:10.109115 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:10.109084 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5412c3_19d3_4d81_b59a_20c482bc8b75.slice/crio-a2374dc7f3ea428eb6465c3e81e54f7952040c479f31c3617eb71d910da19fb4 WatchSource:0}: Error finding container a2374dc7f3ea428eb6465c3e81e54f7952040c479f31c3617eb71d910da19fb4: Status 404 returned error can't find the container with id a2374dc7f3ea428eb6465c3e81e54f7952040c479f31c3617eb71d910da19fb4 Oct 31 03:53:10.111849 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:10.111804 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:53:10.111962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:10.111907 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:53:10.111962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:10.111952 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:53:10.954375 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:10.954329 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" event={"ID":"fa5412c3-19d3-4d81-b59a-20c482bc8b75","Type":"ContainerStarted","Data":"5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7"} Oct 31 03:53:10.954563 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:10.954432 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" event={"ID":"fa5412c3-19d3-4d81-b59a-20c482bc8b75","Type":"ContainerStarted","Data":"a2374dc7f3ea428eb6465c3e81e54f7952040c479f31c3617eb71d910da19fb4"} Oct 31 03:53:10.981354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:10.981320 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:53:11.071854 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.071798 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" podStartSLOduration=2.071781783 podStartE2EDuration="2.071781783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:11.066669895 +0000 UTC m=+1488.067295118" watchObservedRunningTime="2026-10-31 03:53:11.071781783 +0000 UTC m=+1488.072407011" Oct 31 03:53:11.663341 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.663300 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:11.665411 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.665386 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:53:11.665533 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.665444 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:53:11.706720 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.706673 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk"] Oct 31 03:53:11.711864 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.711836 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:11.760948 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.760921 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:53:11.765838 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.765806 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:53:11.765838 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.765806 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:53:11.765838 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.765816 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:53:11.785832 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.785790 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ec6a414-q5fdk\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:11.786017 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.785843 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ec6a414-q5fdk\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:11.786017 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.785938 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fnm8\" (UniqueName: \"kubernetes.io/projected/4308b4d0-e562-48c4-9e6b-d95a23e32d27-kube-api-access-2fnm8\") pod \"maas-api-revoke-sub-0ec6a414-q5fdk\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:11.813789 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.813739 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk"] Oct 31 03:53:11.886883 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.886848 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ec6a414-q5fdk\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:11.887077 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.886896 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ec6a414-q5fdk\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:11.887077 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.886955 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnm8\" (UniqueName: \"kubernetes.io/projected/4308b4d0-e562-48c4-9e6b-d95a23e32d27-kube-api-access-2fnm8\") pod \"maas-api-revoke-sub-0ec6a414-q5fdk\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:11.887656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.887633 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ec6a414-q5fdk\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:11.889360 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.889341 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ec6a414-q5fdk\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:11.950491 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:11.950420 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnm8\" (UniqueName: \"kubernetes.io/projected/4308b4d0-e562-48c4-9e6b-d95a23e32d27-kube-api-access-2fnm8\") pod \"maas-api-revoke-sub-0ec6a414-q5fdk\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:12.055005 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:12.054963 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:12.483675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:12.483633 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk"] Oct 31 03:53:12.488186 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:12.487704 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4308b4d0_e562_48c4_9e6b_d95a23e32d27.slice/crio-725f4461d2fe456d3fb16d7a570d89f1983fac4b3a19643ba65f619363e00075 WatchSource:0}: Error finding container 725f4461d2fe456d3fb16d7a570d89f1983fac4b3a19643ba65f619363e00075: Status 404 returned error can't find the container with id 725f4461d2fe456d3fb16d7a570d89f1983fac4b3a19643ba65f619363e00075 Oct 31 03:53:12.663382 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:12.663337 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:53:12.663890 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:12.663402 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:53:13.481966 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:13.481914 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" event={"ID":"4308b4d0-e562-48c4-9e6b-d95a23e32d27","Type":"ContainerStarted","Data":"725f4461d2fe456d3fb16d7a570d89f1983fac4b3a19643ba65f619363e00075"} Oct 31 03:53:13.665788 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:13.665749 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:53:13.666401 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:13.666365 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:53:13.943149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:13.938610 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:14.094769 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:14.092200 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:14.186100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:14.186067 4426 generic.go:356] "Generic (PLEG): container finished" podID="4308b4d0-e562-48c4-9e6b-d95a23e32d27" containerID="187dc2dc83e0bd9c790bdd549e434a6f3c6245dbe306ecf2df9bf0178a54e86d" exitCode=0 Oct 31 03:53:14.186275 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:14.186193 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" event={"ID":"4308b4d0-e562-48c4-9e6b-d95a23e32d27","Type":"ContainerDied","Data":"187dc2dc83e0bd9c790bdd549e434a6f3c6245dbe306ecf2df9bf0178a54e86d"} Oct 31 03:53:14.665175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:14.662839 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:53:14.665175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:14.662922 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:53:15.059929 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.059843 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs"] Oct 31 03:53:15.101678 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.101645 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d04429-nx5fs"] Oct 31 03:53:15.106529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.106495 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb"] Oct 31 03:53:15.110921 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.110898 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b7447e7-hlxqb"] Oct 31 03:53:15.171591 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.171550 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b11ee5-9421-4de2-8948-4a55e683ba1a" path="/var/lib/kubelet/pods/68b11ee5-9421-4de2-8948-4a55e683ba1a/volumes" Oct 31 03:53:15.172328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.172306 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d528b635-9901-4bc4-8936-58997ca5b278" path="/var/lib/kubelet/pods/d528b635-9901-4bc4-8936-58997ca5b278/volumes" Oct 31 03:53:15.496467 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.496441 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:15.663636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.663184 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:53:15.663636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.663243 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:53:15.674553 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.674512 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4308b4d0-e562-48c4-9e6b-d95a23e32d27-kube-api-access-2fnm8\" (UniqueName: \"kubernetes.io/projected/4308b4d0-e562-48c4-9e6b-d95a23e32d27-kube-api-access-2fnm8\") pod \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " Oct 31 03:53:15.674912 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.674892 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-xks-ca\") pod \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " Oct 31 03:53:15.675315 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.675236 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-openshift-ca\") pod \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\" (UID: \"4308b4d0-e562-48c4-9e6b-d95a23e32d27\") " Oct 31 03:53:15.675701 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.675670 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-openshift-ca" pod "4308b4d0-e562-48c4-9e6b-d95a23e32d27" (UID: "4308b4d0-e562-48c4-9e6b-d95a23e32d27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:15.675953 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.675934 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:15.678348 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.678317 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4308b4d0-e562-48c4-9e6b-d95a23e32d27-kube-api-access-2fnm8" pod "4308b4d0-e562-48c4-9e6b-d95a23e32d27" (UID: "4308b4d0-e562-48c4-9e6b-d95a23e32d27"). InnerVolumeSpecName "kube-api-access-2fnm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:15.682501 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.682458 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-xks-ca" pod "4308b4d0-e562-48c4-9e6b-d95a23e32d27" (UID: "4308b4d0-e562-48c4-9e6b-d95a23e32d27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:15.777263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.777178 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fnm8\" (UniqueName: \"kubernetes.io/projected/4308b4d0-e562-48c4-9e6b-d95a23e32d27-kube-api-access-2fnm8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:15.777263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:15.777209 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4308b4d0-e562-48c4-9e6b-d95a23e32d27-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:16.203774 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:16.203743 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" Oct 31 03:53:16.204252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:16.203742 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk" event={"ID":"4308b4d0-e562-48c4-9e6b-d95a23e32d27","Type":"ContainerDied","Data":"725f4461d2fe456d3fb16d7a570d89f1983fac4b3a19643ba65f619363e00075"} Oct 31 03:53:16.204252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:16.203894 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="725f4461d2fe456d3fb16d7a570d89f1983fac4b3a19643ba65f619363e00075" Oct 31 03:53:16.663014 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:16.662967 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:53:16.663218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:16.663041 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:53:16.971643 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:16.971369 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48"] Oct 31 03:53:16.972513 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:16.972492 4426 cpu_manager.go:414] "containerMap: removing container" podUID="4308b4d0-e562-48c4-9e6b-d95a23e32d27" containerName="revoke-keys" Oct 31 03:53:16.972614 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:16.972604 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4308b4d0-e562-48c4-9e6b-d95a23e32d27" containerName="revoke-keys" Oct 31 03:53:16.972796 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:16.972786 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="4308b4d0-e562-48c4-9e6b-d95a23e32d27" containerName="revoke-keys" Oct 31 03:53:16.976559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:16.976539 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:17.005764 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.005724 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48"] Oct 31 03:53:17.027775 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.027744 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:53:17.044627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.044596 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:53:17.044969 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.044952 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:53:17.045329 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.045268 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:53:17.086422 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.086387 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg"] Oct 31 03:53:17.093881 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.093852 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a90f663-vjzqg"] Oct 31 03:53:17.098241 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.098205 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539e5bc7-5zs48\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:17.098405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.098266 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9f8\" (UniqueName: \"kubernetes.io/projected/0c3f2cef-0a5a-489b-9d4a-29c941286885-kube-api-access-wq9f8\") pod \"maas-api-revoke-sub-539e5bc7-5zs48\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:17.098578 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.098551 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539e5bc7-5zs48\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:17.132327 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.132287 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8482bf-17f1-4a3d-b705-015efeb50fe9" path="/var/lib/kubelet/pods/ec8482bf-17f1-4a3d-b705-015efeb50fe9/volumes" Oct 31 03:53:17.200100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.200055 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539e5bc7-5zs48\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:17.200100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.200105 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9f8\" (UniqueName: \"kubernetes.io/projected/0c3f2cef-0a5a-489b-9d4a-29c941286885-kube-api-access-wq9f8\") pod \"maas-api-revoke-sub-539e5bc7-5zs48\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:17.200384 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.200274 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539e5bc7-5zs48\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:17.200933 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.200906 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539e5bc7-5zs48\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:17.202651 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.202624 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539e5bc7-5zs48\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:17.259177 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.259075 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9f8\" (UniqueName: \"kubernetes.io/projected/0c3f2cef-0a5a-489b-9d4a-29c941286885-kube-api-access-wq9f8\") pod \"maas-api-revoke-sub-539e5bc7-5zs48\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:17.301961 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.301921 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:17.704480 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:17.704445 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48"] Oct 31 03:53:17.710419 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:17.710386 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3f2cef_0a5a_489b_9d4a_29c941286885.slice/crio-39586540306d4762a00f89deb8f43dda030fa2e79ba36854f418118d81f38d74 WatchSource:0}: Error finding container 39586540306d4762a00f89deb8f43dda030fa2e79ba36854f418118d81f38d74: Status 404 returned error can't find the container with id 39586540306d4762a00f89deb8f43dda030fa2e79ba36854f418118d81f38d74 Oct 31 03:53:18.010309 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.010207 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:18.014405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.014266 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:18.019406 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.019377 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 03:53:18.232448 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.232404 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" event={"ID":"0c3f2cef-0a5a-489b-9d4a-29c941286885","Type":"ContainerStarted","Data":"d0bbedf7488a9fcd47a4beffa492ba7293d23e2698786c85d2101d0404062f00"} Oct 31 03:53:18.232628 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.232477 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" event={"ID":"0c3f2cef-0a5a-489b-9d4a-29c941286885","Type":"ContainerStarted","Data":"39586540306d4762a00f89deb8f43dda030fa2e79ba36854f418118d81f38d74"} Oct 31 03:53:18.350629 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.350494 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" podStartSLOduration=2.35047564 podStartE2EDuration="2.35047564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:18.350044885 +0000 UTC m=+1495.350670114" watchObservedRunningTime="2026-10-31 03:53:18.35047564 +0000 UTC m=+1495.351100872" Oct 31 03:53:18.445371 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.445334 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8"] Oct 31 03:53:18.458318 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.458086 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:18.508619 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.508584 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8"] Oct 31 03:53:18.523173 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.522369 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrfdf\" (UniqueName: \"kubernetes.io/projected/df025f27-a4ea-4c5b-a078-1773613c06f3-kube-api-access-wrfdf\") pod \"maas-api-revoke-sub-b9e845eb-x2gt8\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:18.523173 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.522453 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9e845eb-x2gt8\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:18.523173 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.522567 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9e845eb-x2gt8\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:18.623911 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.623805 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9e845eb-x2gt8\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:18.624118 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.623926 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrfdf\" (UniqueName: \"kubernetes.io/projected/df025f27-a4ea-4c5b-a078-1773613c06f3-kube-api-access-wrfdf\") pod \"maas-api-revoke-sub-b9e845eb-x2gt8\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:18.624118 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.623977 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9e845eb-x2gt8\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:18.624752 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.624726 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9e845eb-x2gt8\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:18.626392 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.626362 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9e845eb-x2gt8\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:18.681946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.681915 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrfdf\" (UniqueName: \"kubernetes.io/projected/df025f27-a4ea-4c5b-a078-1773613c06f3-kube-api-access-wrfdf\") pod \"maas-api-revoke-sub-b9e845eb-x2gt8\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:18.773721 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:18.773683 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:19.166986 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:19.166953 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8"] Oct 31 03:53:19.168521 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:19.168478 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf025f27_a4ea_4c5b_a078_1773613c06f3.slice/crio-9be01f3a6ea4f11a645e93afaccbde64785de95c17ae95c5c5fc2354ec084078 WatchSource:0}: Error finding container 9be01f3a6ea4f11a645e93afaccbde64785de95c17ae95c5c5fc2354ec084078: Status 404 returned error can't find the container with id 9be01f3a6ea4f11a645e93afaccbde64785de95c17ae95c5c5fc2354ec084078 Oct 31 03:53:19.243338 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:19.243298 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" event={"ID":"df025f27-a4ea-4c5b-a078-1773613c06f3","Type":"ContainerStarted","Data":"9be01f3a6ea4f11a645e93afaccbde64785de95c17ae95c5c5fc2354ec084078"} Oct 31 03:53:19.245615 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:19.245585 4426 generic.go:356] "Generic (PLEG): container finished" podID="0c3f2cef-0a5a-489b-9d4a-29c941286885" containerID="d0bbedf7488a9fcd47a4beffa492ba7293d23e2698786c85d2101d0404062f00" exitCode=0 Oct 31 03:53:19.245765 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:19.245676 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" event={"ID":"0c3f2cef-0a5a-489b-9d4a-29c941286885","Type":"ContainerDied","Data":"d0bbedf7488a9fcd47a4beffa492ba7293d23e2698786c85d2101d0404062f00"} Oct 31 03:53:20.259181 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.259106 4426 generic.go:356] "Generic (PLEG): container finished" podID="df025f27-a4ea-4c5b-a078-1773613c06f3" containerID="c3a4f7b4d7f1468a00fc65705aa84fcef313e4863db2fa0cb116ae240d3fa69f" exitCode=0 Oct 31 03:53:20.259656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.259192 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" event={"ID":"df025f27-a4ea-4c5b-a078-1773613c06f3","Type":"ContainerDied","Data":"c3a4f7b4d7f1468a00fc65705aa84fcef313e4863db2fa0cb116ae240d3fa69f"} Oct 31 03:53:20.446706 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.446669 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:20.548774 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.548730 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3f2cef-0a5a-489b-9d4a-29c941286885-kube-api-access-wq9f8\" (UniqueName: \"kubernetes.io/projected/0c3f2cef-0a5a-489b-9d4a-29c941286885-kube-api-access-wq9f8\") pod \"0c3f2cef-0a5a-489b-9d4a-29c941286885\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " Oct 31 03:53:20.548980 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.548788 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-openshift-ca\") pod \"0c3f2cef-0a5a-489b-9d4a-29c941286885\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " Oct 31 03:53:20.548980 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.548823 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-xks-ca\") pod \"0c3f2cef-0a5a-489b-9d4a-29c941286885\" (UID: \"0c3f2cef-0a5a-489b-9d4a-29c941286885\") " Oct 31 03:53:20.549331 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.549295 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-openshift-ca" pod "0c3f2cef-0a5a-489b-9d4a-29c941286885" (UID: "0c3f2cef-0a5a-489b-9d4a-29c941286885"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:20.549589 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.549562 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:20.551240 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.551210 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-xks-ca" pod "0c3f2cef-0a5a-489b-9d4a-29c941286885" (UID: "0c3f2cef-0a5a-489b-9d4a-29c941286885"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:20.551240 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.551228 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3f2cef-0a5a-489b-9d4a-29c941286885-kube-api-access-wq9f8" pod "0c3f2cef-0a5a-489b-9d4a-29c941286885" (UID: "0c3f2cef-0a5a-489b-9d4a-29c941286885"). InnerVolumeSpecName "kube-api-access-wq9f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:20.651233 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.651186 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9f8\" (UniqueName: \"kubernetes.io/projected/0c3f2cef-0a5a-489b-9d4a-29c941286885-kube-api-access-wq9f8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:20.651233 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:20.651231 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c3f2cef-0a5a-489b-9d4a-29c941286885-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:21.270928 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.270896 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" Oct 31 03:53:21.271436 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.270931 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48" event={"ID":"0c3f2cef-0a5a-489b-9d4a-29c941286885","Type":"ContainerDied","Data":"39586540306d4762a00f89deb8f43dda030fa2e79ba36854f418118d81f38d74"} Oct 31 03:53:21.271436 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.270982 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39586540306d4762a00f89deb8f43dda030fa2e79ba36854f418118d81f38d74" Oct 31 03:53:21.445434 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.445399 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:21.465938 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.465903 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-xks-ca\") pod \"df025f27-a4ea-4c5b-a078-1773613c06f3\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " Oct 31 03:53:21.466105 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.465970 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-openshift-ca\") pod \"df025f27-a4ea-4c5b-a078-1773613c06f3\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " Oct 31 03:53:21.466105 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.466013 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df025f27-a4ea-4c5b-a078-1773613c06f3-kube-api-access-wrfdf\" (UniqueName: \"kubernetes.io/projected/df025f27-a4ea-4c5b-a078-1773613c06f3-kube-api-access-wrfdf\") pod \"df025f27-a4ea-4c5b-a078-1773613c06f3\" (UID: \"df025f27-a4ea-4c5b-a078-1773613c06f3\") " Oct 31 03:53:21.466340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.466316 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-openshift-ca" pod "df025f27-a4ea-4c5b-a078-1773613c06f3" (UID: "df025f27-a4ea-4c5b-a078-1773613c06f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:21.466532 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.466517 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:21.468270 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.468241 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-xks-ca" pod "df025f27-a4ea-4c5b-a078-1773613c06f3" (UID: "df025f27-a4ea-4c5b-a078-1773613c06f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:21.468270 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.468253 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df025f27-a4ea-4c5b-a078-1773613c06f3-kube-api-access-wrfdf" pod "df025f27-a4ea-4c5b-a078-1773613c06f3" (UID: "df025f27-a4ea-4c5b-a078-1773613c06f3"). InnerVolumeSpecName "kube-api-access-wrfdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:21.567414 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.567379 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df025f27-a4ea-4c5b-a078-1773613c06f3-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:21.567414 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:21.567411 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrfdf\" (UniqueName: \"kubernetes.io/projected/df025f27-a4ea-4c5b-a078-1773613c06f3-kube-api-access-wrfdf\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:22.281874 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:22.281842 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" Oct 31 03:53:22.281874 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:22.281858 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8" event={"ID":"df025f27-a4ea-4c5b-a078-1773613c06f3","Type":"ContainerDied","Data":"9be01f3a6ea4f11a645e93afaccbde64785de95c17ae95c5c5fc2354ec084078"} Oct 31 03:53:22.282341 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:22.281905 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9be01f3a6ea4f11a645e93afaccbde64785de95c17ae95c5c5fc2354ec084078" Oct 31 03:53:23.225042 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.224996 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc"] Oct 31 03:53:23.226054 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.226028 4426 cpu_manager.go:414] "containerMap: removing container" podUID="0c3f2cef-0a5a-489b-9d4a-29c941286885" containerName="revoke-keys" Oct 31 03:53:23.226156 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.226057 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c3f2cef-0a5a-489b-9d4a-29c941286885" containerName="revoke-keys" Oct 31 03:53:23.226156 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.226075 4426 cpu_manager.go:414] "containerMap: removing container" podUID="df025f27-a4ea-4c5b-a078-1773613c06f3" containerName="revoke-keys" Oct 31 03:53:23.226156 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.226083 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df025f27-a4ea-4c5b-a078-1773613c06f3" containerName="revoke-keys" Oct 31 03:53:23.226318 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.226267 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c3f2cef-0a5a-489b-9d4a-29c941286885" containerName="revoke-keys" Oct 31 03:53:23.226318 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.226286 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="df025f27-a4ea-4c5b-a078-1773613c06f3" containerName="revoke-keys" Oct 31 03:53:23.230857 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.230833 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:23.282208 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.282170 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc"] Oct 31 03:53:23.293056 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.293018 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c55df56-zqfbc\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:23.293424 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.293392 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c55df56-zqfbc\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:23.293613 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.293587 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2zfc\" (UniqueName: \"kubernetes.io/projected/7e0cb792-c5b3-430f-bd14-9572930df880-kube-api-access-z2zfc\") pod \"maas-api-revoke-sub-8c55df56-zqfbc\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:23.321724 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.321694 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:53:23.326202 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.326157 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:53:23.326371 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.326263 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:53:23.326371 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.326334 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:53:23.349611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.349583 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:53:23.354293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.354266 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:53:23.395320 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.395271 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2zfc\" (UniqueName: \"kubernetes.io/projected/7e0cb792-c5b3-430f-bd14-9572930df880-kube-api-access-z2zfc\") pod \"maas-api-revoke-sub-8c55df56-zqfbc\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:23.395519 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.395399 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c55df56-zqfbc\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:23.395519 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.395498 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c55df56-zqfbc\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:23.396157 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.396097 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c55df56-zqfbc\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:23.401590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.401555 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c55df56-zqfbc\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:23.453281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.453250 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2zfc\" (UniqueName: \"kubernetes.io/projected/7e0cb792-c5b3-430f-bd14-9572930df880-kube-api-access-z2zfc\") pod \"maas-api-revoke-sub-8c55df56-zqfbc\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:23.457190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.457156 4426 scope.go:122] "RemoveContainer" containerID="fdac12a574aba06f3aa2daf0c9f07ffe7a3f52840d579aa04ed59fd03517bec3" Oct 31 03:53:23.472577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.472554 4426 scope.go:122] "RemoveContainer" containerID="7916937100f423512b0dc8e4afd1f09bf5aee29cffbb63c0751315db775b9c14" Oct 31 03:53:23.483518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.483498 4426 scope.go:122] "RemoveContainer" containerID="342d02f6c0432dbcab5bfa9c68b8702eb3721396517dd467f06e795611524fd6" Oct 31 03:53:23.493722 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.493700 4426 scope.go:122] "RemoveContainer" containerID="ee851c95732e5317e880b35422639db8d1b231a398d59a40f01ce9275cc6784b" Oct 31 03:53:23.504078 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.504054 4426 scope.go:122] "RemoveContainer" containerID="64db89033956adbf471421cedcb40b7c3a8fa8d747510f3fda4252b7626fd40d" Oct 31 03:53:23.516736 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.516711 4426 scope.go:122] "RemoveContainer" containerID="530bdea53a6c57efe59e4becce555146719607fa1baac2be4a268c329e93e01b" Oct 31 03:53:23.549491 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.549468 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:23.946067 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:23.946044 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc"] Oct 31 03:53:23.948912 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:23.948886 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0cb792_c5b3_430f_bd14_9572930df880.slice/crio-101f494715cc4a7f17c7f54bfd258cbe209ec9581d4695fd2c9f00613ac4a2d6 WatchSource:0}: Error finding container 101f494715cc4a7f17c7f54bfd258cbe209ec9581d4695fd2c9f00613ac4a2d6: Status 404 returned error can't find the container with id 101f494715cc4a7f17c7f54bfd258cbe209ec9581d4695fd2c9f00613ac4a2d6 Oct 31 03:53:24.312704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:24.312616 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" event={"ID":"7e0cb792-c5b3-430f-bd14-9572930df880","Type":"ContainerStarted","Data":"101f494715cc4a7f17c7f54bfd258cbe209ec9581d4695fd2c9f00613ac4a2d6"} Oct 31 03:53:25.047726 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:25.047682 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7"] Oct 31 03:53:25.093713 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:25.093680 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f0902d-8fqf7"] Oct 31 03:53:25.132284 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:25.132209 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045b5abd-acde-4a56-8e6e-aecb2f7de383" path="/var/lib/kubelet/pods/045b5abd-acde-4a56-8e6e-aecb2f7de383/volumes" Oct 31 03:53:25.325140 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:25.325088 4426 generic.go:356] "Generic (PLEG): container finished" podID="7e0cb792-c5b3-430f-bd14-9572930df880" containerID="c06394432d110b463f5516da73cf228b1631d0da4339da75abf5253a9ce2b4a2" exitCode=0 Oct 31 03:53:25.325522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:25.325178 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" event={"ID":"7e0cb792-c5b3-430f-bd14-9572930df880","Type":"ContainerDied","Data":"c06394432d110b463f5516da73cf228b1631d0da4339da75abf5253a9ce2b4a2"} Oct 31 03:53:26.503994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:26.503961 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:26.536687 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:26.536649 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0cb792-c5b3-430f-bd14-9572930df880-kube-api-access-z2zfc\" (UniqueName: \"kubernetes.io/projected/7e0cb792-c5b3-430f-bd14-9572930df880-kube-api-access-z2zfc\") pod \"7e0cb792-c5b3-430f-bd14-9572930df880\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " Oct 31 03:53:26.536904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:26.536882 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-xks-ca\") pod \"7e0cb792-c5b3-430f-bd14-9572930df880\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " Oct 31 03:53:26.536963 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:26.536936 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-openshift-ca\") pod \"7e0cb792-c5b3-430f-bd14-9572930df880\" (UID: \"7e0cb792-c5b3-430f-bd14-9572930df880\") " Oct 31 03:53:26.537379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:26.537337 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-openshift-ca" pod "7e0cb792-c5b3-430f-bd14-9572930df880" (UID: "7e0cb792-c5b3-430f-bd14-9572930df880"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:26.537655 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:26.537630 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:26.539078 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:26.539046 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-xks-ca" pod "7e0cb792-c5b3-430f-bd14-9572930df880" (UID: "7e0cb792-c5b3-430f-bd14-9572930df880"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:26.539281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:26.539259 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0cb792-c5b3-430f-bd14-9572930df880-kube-api-access-z2zfc" pod "7e0cb792-c5b3-430f-bd14-9572930df880" (UID: "7e0cb792-c5b3-430f-bd14-9572930df880"). InnerVolumeSpecName "kube-api-access-z2zfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:26.638858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:26.638771 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0cb792-c5b3-430f-bd14-9572930df880-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:26.638858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:26.638809 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2zfc\" (UniqueName: \"kubernetes.io/projected/7e0cb792-c5b3-430f-bd14-9572930df880-kube-api-access-z2zfc\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:27.345962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:27.345930 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" Oct 31 03:53:27.346186 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:27.345961 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc" event={"ID":"7e0cb792-c5b3-430f-bd14-9572930df880","Type":"ContainerDied","Data":"101f494715cc4a7f17c7f54bfd258cbe209ec9581d4695fd2c9f00613ac4a2d6"} Oct 31 03:53:27.346186 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:27.346010 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="101f494715cc4a7f17c7f54bfd258cbe209ec9581d4695fd2c9f00613ac4a2d6" Oct 31 03:53:28.048353 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:28.048314 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx"] Oct 31 03:53:28.097533 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:28.097496 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a24e579-ssssx"] Oct 31 03:53:29.132359 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:29.132324 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8147a502-9522-4659-b4d0-aea5b2b9239b" path="/var/lib/kubelet/pods/8147a502-9522-4659-b4d0-aea5b2b9239b/volumes" Oct 31 03:53:31.048895 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.048857 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg"] Oct 31 03:53:31.094571 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.094538 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c621e4b1-qlwwg"] Oct 31 03:53:31.132835 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.132796 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="802cbbcb-0123-49f5-80c3-44dfb109cf5d" path="/var/lib/kubelet/pods/802cbbcb-0123-49f5-80c3-44dfb109cf5d/volumes" Oct 31 03:53:31.717829 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.717795 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc"] Oct 31 03:53:31.718551 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.718528 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7e0cb792-c5b3-430f-bd14-9572930df880" containerName="revoke-keys" Oct 31 03:53:31.718551 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.718551 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0cb792-c5b3-430f-bd14-9572930df880" containerName="revoke-keys" Oct 31 03:53:31.718751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.718719 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0cb792-c5b3-430f-bd14-9572930df880" containerName="revoke-keys" Oct 31 03:53:31.723883 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.723859 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.773039 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.773005 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d7b7bc-openshift-default-dockercfg-nszpp\"" Oct 31 03:53:31.811404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.811369 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.811573 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.811415 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbn8\" (UniqueName: \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-kube-api-access-ljbn8\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.811573 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.811514 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-envoy\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.811573 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.811541 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-data\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.812060 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.811582 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-certs\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.812060 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.811706 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-podinfo\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.812060 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.811763 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-token\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.812060 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.811852 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istiod-ca-cert\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.812060 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.812039 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-credential-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.825237 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.825206 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc"] Oct 31 03:53:31.913721 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.913677 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-credential-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.913903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.913739 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.913903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.913769 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbn8\" (UniqueName: \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-kube-api-access-ljbn8\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.913903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.913801 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-envoy\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.914062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.913943 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-data\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.914062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.914016 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-certs\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.914218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.914082 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-podinfo\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.914218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.914109 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-credential-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.914218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.914168 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-token\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.914382 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.914230 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.914382 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.914249 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istiod-ca-cert\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.914382 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.914284 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-data\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.914544 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.914411 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-certs\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.914903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.914883 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istiod-ca-cert\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.916289 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.916269 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-envoy\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.916590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.916571 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-podinfo\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.971487 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.971421 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-token\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:31.971653 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:31.971629 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbn8\" (UniqueName: \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-kube-api-access-ljbn8\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:32.037667 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:32.037634 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:32.048334 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:32.048298 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv"] Oct 31 03:53:32.053188 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:32.053161 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6ae32c-zjhfv"] Oct 31 03:53:32.477958 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:32.477912 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc"] Oct 31 03:53:32.480963 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:32.480933 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9ad3b0_b2d6_4c1b_b097_d15072fd93aa.slice/crio-3d1ea5690110b30d8151e91318845d9afd3fdeb5d20f587f521b9cf609d7eba2 WatchSource:0}: Error finding container 3d1ea5690110b30d8151e91318845d9afd3fdeb5d20f587f521b9cf609d7eba2: Status 404 returned error can't find the container with id 3d1ea5690110b30d8151e91318845d9afd3fdeb5d20f587f521b9cf609d7eba2 Oct 31 03:53:32.483798 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:32.483762 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:53:32.483914 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:32.483836 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:53:32.483914 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:32.483867 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:53:32.859628 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:53:32.859585 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:53:33.150367 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:33.150335 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ea6b9d-9837-485b-aba6-e75eb17df90b" path="/var/lib/kubelet/pods/86ea6b9d-9837-485b-aba6-e75eb17df90b/volumes" Oct 31 03:53:33.398206 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:33.398159 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" event={"ID":"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa","Type":"ContainerStarted","Data":"17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8"} Oct 31 03:53:33.398378 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:33.398235 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" event={"ID":"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa","Type":"ContainerStarted","Data":"3d1ea5690110b30d8151e91318845d9afd3fdeb5d20f587f521b9cf609d7eba2"} Oct 31 03:53:33.518210 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:33.514351 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" podStartSLOduration=2.514330289 podStartE2EDuration="2.514330289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:33.512468999 +0000 UTC m=+1510.513094228" watchObservedRunningTime="2026-10-31 03:53:33.514330289 +0000 UTC m=+1510.514955517" Oct 31 03:53:34.042146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:34.041388 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:34.042146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:34.041538 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:53:34.042146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:34.041611 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" podUID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:53:35.041150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:35.040525 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:53:35.041150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:35.040589 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" podUID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:53:36.039151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:36.038552 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:53:36.039151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:36.038616 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" podUID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:53:37.039151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:37.038557 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:53:37.039151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:37.038619 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" podUID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:53:38.042301 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:38.042259 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:53:38.042892 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:38.042856 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" podUID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:53:39.038632 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:39.038309 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:53:39.038632 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:39.038368 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" podUID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:53:40.054150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:40.053846 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:40.055537 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:40.055508 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:40.057195 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:40.057173 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 03:53:42.048033 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:42.047995 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z"] Oct 31 03:53:42.094925 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:42.094890 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8fae9e4-qfr8z"] Oct 31 03:53:43.203193 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:43.203156 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334d46da-9dcf-45c1-9709-14200152e8b4" path="/var/lib/kubelet/pods/334d46da-9dcf-45c1-9709-14200152e8b4/volumes" Oct 31 03:53:44.153380 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:44.153310 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx"] Oct 31 03:53:44.190232 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:44.190196 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9650182-6klnx"] Oct 31 03:53:44.296289 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:44.296236 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:44.314695 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:44.314629 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:44.402152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:44.399527 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:44.453218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:44.453092 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:44.514297 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:44.514246 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:44.590149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:44.589409 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:44.728596 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:44.728473 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:44.792764 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:44.792356 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Oct 31 03:53:44.930147 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:44.929714 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:45.152383 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:45.152342 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298bde85-443f-4237-8e71-7737b7630618" path="/var/lib/kubelet/pods/298bde85-443f-4237-8e71-7737b7630618/volumes" Oct 31 03:53:46.049966 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:46.049928 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj"] Oct 31 03:53:46.094881 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:46.094837 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53c9f57-j5gfj"] Oct 31 03:53:47.132705 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.132662 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ea1f60-1ffe-43e3-899b-6925af1d0608" path="/var/lib/kubelet/pods/a3ea1f60-1ffe-43e3-899b-6925af1d0608/volumes" Oct 31 03:53:47.263246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.263201 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt"] Oct 31 03:53:47.268087 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.268064 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:47.303272 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.303236 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e98011-pjvdt\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:47.303445 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.303279 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e98011-pjvdt\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:47.303445 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.303413 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgj9\" (UniqueName: \"kubernetes.io/projected/a35eecf8-6ad3-42d3-9144-85ce3df82f32-kube-api-access-gzgj9\") pod \"maas-api-revoke-sub-46e98011-pjvdt\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:47.363977 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.363941 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:53:47.364291 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.363956 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:53:47.364291 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.363989 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:53:47.364291 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.364271 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:53:47.369458 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.369426 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt"] Oct 31 03:53:47.404442 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.404406 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e98011-pjvdt\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:47.404625 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.404499 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgj9\" (UniqueName: \"kubernetes.io/projected/a35eecf8-6ad3-42d3-9144-85ce3df82f32-kube-api-access-gzgj9\") pod \"maas-api-revoke-sub-46e98011-pjvdt\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:47.404680 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.404659 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e98011-pjvdt\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:47.405312 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.405282 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e98011-pjvdt\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:47.407018 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.406993 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e98011-pjvdt\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:47.462980 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.462949 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgj9\" (UniqueName: \"kubernetes.io/projected/a35eecf8-6ad3-42d3-9144-85ce3df82f32-kube-api-access-gzgj9\") pod \"maas-api-revoke-sub-46e98011-pjvdt\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:47.583068 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.583031 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:47.973633 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:47.973554 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt"] Oct 31 03:53:47.977156 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:47.977094 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35eecf8_6ad3_42d3_9144_85ce3df82f32.slice/crio-03333ed4e948c33ecd182d40d0173412eb2e86aa8814bfd50e0a64470673e2ce WatchSource:0}: Error finding container 03333ed4e948c33ecd182d40d0173412eb2e86aa8814bfd50e0a64470673e2ce: Status 404 returned error can't find the container with id 03333ed4e948c33ecd182d40d0173412eb2e86aa8814bfd50e0a64470673e2ce Oct 31 03:53:48.649789 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:48.649746 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" event={"ID":"a35eecf8-6ad3-42d3-9144-85ce3df82f32","Type":"ContainerStarted","Data":"6f5f0f148f411b269dff5e1d9b9bd22426cb72ce822eefca5c760b8e0ee9ffe9"} Oct 31 03:53:48.650362 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:48.649846 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" event={"ID":"a35eecf8-6ad3-42d3-9144-85ce3df82f32","Type":"ContainerStarted","Data":"03333ed4e948c33ecd182d40d0173412eb2e86aa8814bfd50e0a64470673e2ce"} Oct 31 03:53:48.759070 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:48.759002 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" podStartSLOduration=1.7589829080000001 podStartE2EDuration="1.758982908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:48.757841573 +0000 UTC m=+1525.758466802" watchObservedRunningTime="2026-10-31 03:53:48.758982908 +0000 UTC m=+1525.759608137" Oct 31 03:53:48.834165 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:48.834104 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6"] Oct 31 03:53:48.838526 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:48.838501 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:48.894621 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:48.894581 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6"] Oct 31 03:53:48.941203 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:48.938928 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtnmc\" (UniqueName: \"kubernetes.io/projected/2503d2fb-448a-48e3-96cf-c37f438101ea-kube-api-access-wtnmc\") pod \"maas-api-revoke-sub-83fe6a26-rbsk6\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:48.941203 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:48.939007 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83fe6a26-rbsk6\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:48.941203 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:48.939174 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83fe6a26-rbsk6\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:49.041025 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:49.040977 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtnmc\" (UniqueName: \"kubernetes.io/projected/2503d2fb-448a-48e3-96cf-c37f438101ea-kube-api-access-wtnmc\") pod \"maas-api-revoke-sub-83fe6a26-rbsk6\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:49.041265 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:49.041069 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83fe6a26-rbsk6\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:49.041265 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:49.041240 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83fe6a26-rbsk6\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:49.041925 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:49.041895 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83fe6a26-rbsk6\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:49.043532 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:49.043508 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83fe6a26-rbsk6\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:49.099220 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:49.099187 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtnmc\" (UniqueName: \"kubernetes.io/projected/2503d2fb-448a-48e3-96cf-c37f438101ea-kube-api-access-wtnmc\") pod \"maas-api-revoke-sub-83fe6a26-rbsk6\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:49.155480 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:49.155449 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:49.545041 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:49.544955 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6"] Oct 31 03:53:49.548790 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:49.548756 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2503d2fb_448a_48e3_96cf_c37f438101ea.slice/crio-3fd5aa287583aaed9f2f8096dcdbe7c39949d9393ec26299b0636df15cacdf75 WatchSource:0}: Error finding container 3fd5aa287583aaed9f2f8096dcdbe7c39949d9393ec26299b0636df15cacdf75: Status 404 returned error can't find the container with id 3fd5aa287583aaed9f2f8096dcdbe7c39949d9393ec26299b0636df15cacdf75 Oct 31 03:53:49.662214 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:49.662165 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" event={"ID":"2503d2fb-448a-48e3-96cf-c37f438101ea","Type":"ContainerStarted","Data":"3fd5aa287583aaed9f2f8096dcdbe7c39949d9393ec26299b0636df15cacdf75"} Oct 31 03:53:49.664162 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:49.664120 4426 generic.go:356] "Generic (PLEG): container finished" podID="a35eecf8-6ad3-42d3-9144-85ce3df82f32" containerID="6f5f0f148f411b269dff5e1d9b9bd22426cb72ce822eefca5c760b8e0ee9ffe9" exitCode=0 Oct 31 03:53:49.664322 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:49.664227 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" event={"ID":"a35eecf8-6ad3-42d3-9144-85ce3df82f32","Type":"ContainerDied","Data":"6f5f0f148f411b269dff5e1d9b9bd22426cb72ce822eefca5c760b8e0ee9ffe9"} Oct 31 03:53:50.685452 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:50.685410 4426 generic.go:356] "Generic (PLEG): container finished" podID="2503d2fb-448a-48e3-96cf-c37f438101ea" containerID="e7bdcd78cf46e25ad06a07a842b7afdc4950165bde83c8ebc5dcd8a070a44d84" exitCode=0 Oct 31 03:53:50.685862 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:50.685497 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" event={"ID":"2503d2fb-448a-48e3-96cf-c37f438101ea","Type":"ContainerDied","Data":"e7bdcd78cf46e25ad06a07a842b7afdc4950165bde83c8ebc5dcd8a070a44d84"} Oct 31 03:53:50.893244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:50.893203 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:50.972883 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:50.972840 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-openshift-ca\") pod \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " Oct 31 03:53:50.973082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:50.972974 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-xks-ca\") pod \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " Oct 31 03:53:50.973082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:50.973048 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a35eecf8-6ad3-42d3-9144-85ce3df82f32-kube-api-access-gzgj9\" (UniqueName: \"kubernetes.io/projected/a35eecf8-6ad3-42d3-9144-85ce3df82f32-kube-api-access-gzgj9\") pod \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\" (UID: \"a35eecf8-6ad3-42d3-9144-85ce3df82f32\") " Oct 31 03:53:50.973350 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:50.973323 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-openshift-ca" pod "a35eecf8-6ad3-42d3-9144-85ce3df82f32" (UID: "a35eecf8-6ad3-42d3-9144-85ce3df82f32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:50.973595 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:50.973577 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:50.975210 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:50.975181 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-xks-ca" pod "a35eecf8-6ad3-42d3-9144-85ce3df82f32" (UID: "a35eecf8-6ad3-42d3-9144-85ce3df82f32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:50.975322 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:50.975299 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35eecf8-6ad3-42d3-9144-85ce3df82f32-kube-api-access-gzgj9" pod "a35eecf8-6ad3-42d3-9144-85ce3df82f32" (UID: "a35eecf8-6ad3-42d3-9144-85ce3df82f32"). InnerVolumeSpecName "kube-api-access-gzgj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:51.074535 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.074428 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzgj9\" (UniqueName: \"kubernetes.io/projected/a35eecf8-6ad3-42d3-9144-85ce3df82f32-kube-api-access-gzgj9\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:51.074535 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.074477 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a35eecf8-6ad3-42d3-9144-85ce3df82f32-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:51.695731 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.695697 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" Oct 31 03:53:51.695731 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.695715 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt" event={"ID":"a35eecf8-6ad3-42d3-9144-85ce3df82f32","Type":"ContainerDied","Data":"03333ed4e948c33ecd182d40d0173412eb2e86aa8814bfd50e0a64470673e2ce"} Oct 31 03:53:51.696271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.695801 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03333ed4e948c33ecd182d40d0173412eb2e86aa8814bfd50e0a64470673e2ce" Oct 31 03:53:51.875029 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.874998 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:51.919861 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.919826 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph"] Oct 31 03:53:51.920529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.920507 4426 cpu_manager.go:414] "containerMap: removing container" podUID="2503d2fb-448a-48e3-96cf-c37f438101ea" containerName="revoke-keys" Oct 31 03:53:51.920529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.920528 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2503d2fb-448a-48e3-96cf-c37f438101ea" containerName="revoke-keys" Oct 31 03:53:51.920719 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.920545 4426 cpu_manager.go:414] "containerMap: removing container" podUID="a35eecf8-6ad3-42d3-9144-85ce3df82f32" containerName="revoke-keys" Oct 31 03:53:51.920719 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.920551 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a35eecf8-6ad3-42d3-9144-85ce3df82f32" containerName="revoke-keys" Oct 31 03:53:51.920719 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.920709 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="2503d2fb-448a-48e3-96cf-c37f438101ea" containerName="revoke-keys" Oct 31 03:53:51.920719 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.920722 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="a35eecf8-6ad3-42d3-9144-85ce3df82f32" containerName="revoke-keys" Oct 31 03:53:51.924246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.924227 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:51.984076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.983997 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2503d2fb-448a-48e3-96cf-c37f438101ea-kube-api-access-wtnmc\" (UniqueName: \"kubernetes.io/projected/2503d2fb-448a-48e3-96cf-c37f438101ea-kube-api-access-wtnmc\") pod \"2503d2fb-448a-48e3-96cf-c37f438101ea\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " Oct 31 03:53:51.984290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.984255 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-xks-ca\") pod \"2503d2fb-448a-48e3-96cf-c37f438101ea\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " Oct 31 03:53:51.984400 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.984376 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-openshift-ca\") pod \"2503d2fb-448a-48e3-96cf-c37f438101ea\" (UID: \"2503d2fb-448a-48e3-96cf-c37f438101ea\") " Oct 31 03:53:51.984663 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.984635 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba7eb0a-x8bph\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:51.985171 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.984777 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba7eb0a-x8bph\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:51.985171 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.984958 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktms\" (UniqueName: \"kubernetes.io/projected/9225d75c-1d9e-45e0-a415-34d8b60be54b-kube-api-access-7ktms\") pod \"maas-api-revoke-sub-7ba7eb0a-x8bph\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:51.985171 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.985086 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph"] Oct 31 03:53:51.985736 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:51.985713 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-openshift-ca" pod "2503d2fb-448a-48e3-96cf-c37f438101ea" (UID: "2503d2fb-448a-48e3-96cf-c37f438101ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:52.004829 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.004785 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2503d2fb-448a-48e3-96cf-c37f438101ea-kube-api-access-wtnmc" pod "2503d2fb-448a-48e3-96cf-c37f438101ea" (UID: "2503d2fb-448a-48e3-96cf-c37f438101ea"). InnerVolumeSpecName "kube-api-access-wtnmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:52.010711 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.010667 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-xks-ca" pod "2503d2fb-448a-48e3-96cf-c37f438101ea" (UID: "2503d2fb-448a-48e3-96cf-c37f438101ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:52.085795 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.085756 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktms\" (UniqueName: \"kubernetes.io/projected/9225d75c-1d9e-45e0-a415-34d8b60be54b-kube-api-access-7ktms\") pod \"maas-api-revoke-sub-7ba7eb0a-x8bph\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:52.085999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.085858 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba7eb0a-x8bph\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:52.085999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.085928 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba7eb0a-x8bph\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:52.086121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.086031 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:52.086121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.086050 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2503d2fb-448a-48e3-96cf-c37f438101ea-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:52.086121 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.086065 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtnmc\" (UniqueName: \"kubernetes.io/projected/2503d2fb-448a-48e3-96cf-c37f438101ea-kube-api-access-wtnmc\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:52.086653 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.086629 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba7eb0a-x8bph\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:52.088566 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.088529 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba7eb0a-x8bph\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:52.144300 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.144264 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktms\" (UniqueName: \"kubernetes.io/projected/9225d75c-1d9e-45e0-a415-34d8b60be54b-kube-api-access-7ktms\") pod \"maas-api-revoke-sub-7ba7eb0a-x8bph\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:52.240163 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.240056 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:52.622771 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.622687 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph"] Oct 31 03:53:52.626744 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:52.626707 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9225d75c_1d9e_45e0_a415_34d8b60be54b.slice/crio-c64efca5d70e0a71e989a6067425a8f7db0649be45cea1442095afec11cb338e WatchSource:0}: Error finding container c64efca5d70e0a71e989a6067425a8f7db0649be45cea1442095afec11cb338e: Status 404 returned error can't find the container with id c64efca5d70e0a71e989a6067425a8f7db0649be45cea1442095afec11cb338e Oct 31 03:53:52.706295 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.706264 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" Oct 31 03:53:52.706698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.706292 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6" event={"ID":"2503d2fb-448a-48e3-96cf-c37f438101ea","Type":"ContainerDied","Data":"3fd5aa287583aaed9f2f8096dcdbe7c39949d9393ec26299b0636df15cacdf75"} Oct 31 03:53:52.706698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.706356 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fd5aa287583aaed9f2f8096dcdbe7c39949d9393ec26299b0636df15cacdf75" Oct 31 03:53:52.708077 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:52.708050 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" event={"ID":"9225d75c-1d9e-45e0-a415-34d8b60be54b","Type":"ContainerStarted","Data":"c64efca5d70e0a71e989a6067425a8f7db0649be45cea1442095afec11cb338e"} Oct 31 03:53:53.656912 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.656871 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw"] Oct 31 03:53:53.661581 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.661559 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:53.717049 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.717013 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw"] Oct 31 03:53:53.740119 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.740072 4426 generic.go:356] "Generic (PLEG): container finished" podID="9225d75c-1d9e-45e0-a415-34d8b60be54b" containerID="93dd93270a423cc9e5e3354c062a75e2bfd7b2456c9bd9470d19b96d2a9e8c96" exitCode=0 Oct 31 03:53:53.740431 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.740237 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" event={"ID":"9225d75c-1d9e-45e0-a415-34d8b60be54b","Type":"ContainerDied","Data":"93dd93270a423cc9e5e3354c062a75e2bfd7b2456c9bd9470d19b96d2a9e8c96"} Oct 31 03:53:53.813872 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.813778 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v497\" (UniqueName: \"kubernetes.io/projected/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-kube-api-access-4v497\") pod \"maas-api-revoke-sub-fbdc3ef4-mfmrw\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:53.814031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.813901 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbdc3ef4-mfmrw\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:53.814031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.814002 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbdc3ef4-mfmrw\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:53.915233 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.915196 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v497\" (UniqueName: \"kubernetes.io/projected/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-kube-api-access-4v497\") pod \"maas-api-revoke-sub-fbdc3ef4-mfmrw\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:53.915430 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.915293 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbdc3ef4-mfmrw\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:53.915430 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.915325 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbdc3ef4-mfmrw\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:53.915966 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.915941 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbdc3ef4-mfmrw\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:53.917755 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.917722 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbdc3ef4-mfmrw\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:53.978030 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:53.977994 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v497\" (UniqueName: \"kubernetes.io/projected/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-kube-api-access-4v497\") pod \"maas-api-revoke-sub-fbdc3ef4-mfmrw\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:54.276476 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:54.276437 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:54.664510 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:54.664475 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw"] Oct 31 03:53:54.668670 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:54.668635 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98d149f_1eec_4e20_bcd7_2a4d7a89bb12.slice/crio-cc17060e62a5bf3897b1f5acd3df0896b5cd003a0670261e37c51a1fd9bdc8bb WatchSource:0}: Error finding container cc17060e62a5bf3897b1f5acd3df0896b5cd003a0670261e37c51a1fd9bdc8bb: Status 404 returned error can't find the container with id cc17060e62a5bf3897b1f5acd3df0896b5cd003a0670261e37c51a1fd9bdc8bb Oct 31 03:53:54.752096 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:54.752046 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" event={"ID":"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12","Type":"ContainerStarted","Data":"cc17060e62a5bf3897b1f5acd3df0896b5cd003a0670261e37c51a1fd9bdc8bb"} Oct 31 03:53:54.913171 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:54.913140 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:55.030481 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.030446 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9225d75c-1d9e-45e0-a415-34d8b60be54b-kube-api-access-7ktms\" (UniqueName: \"kubernetes.io/projected/9225d75c-1d9e-45e0-a415-34d8b60be54b-kube-api-access-7ktms\") pod \"9225d75c-1d9e-45e0-a415-34d8b60be54b\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " Oct 31 03:53:55.030705 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.030687 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-openshift-ca\") pod \"9225d75c-1d9e-45e0-a415-34d8b60be54b\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " Oct 31 03:53:55.030780 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.030718 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-xks-ca\") pod \"9225d75c-1d9e-45e0-a415-34d8b60be54b\" (UID: \"9225d75c-1d9e-45e0-a415-34d8b60be54b\") " Oct 31 03:53:55.031119 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.031094 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-openshift-ca" pod "9225d75c-1d9e-45e0-a415-34d8b60be54b" (UID: "9225d75c-1d9e-45e0-a415-34d8b60be54b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:55.031323 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.031307 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:55.033323 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.033288 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-xks-ca" pod "9225d75c-1d9e-45e0-a415-34d8b60be54b" (UID: "9225d75c-1d9e-45e0-a415-34d8b60be54b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:55.033455 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.033430 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9225d75c-1d9e-45e0-a415-34d8b60be54b-kube-api-access-7ktms" pod "9225d75c-1d9e-45e0-a415-34d8b60be54b" (UID: "9225d75c-1d9e-45e0-a415-34d8b60be54b"). InnerVolumeSpecName "kube-api-access-7ktms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:55.132426 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.132397 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktms\" (UniqueName: \"kubernetes.io/projected/9225d75c-1d9e-45e0-a415-34d8b60be54b-kube-api-access-7ktms\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:55.132426 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.132423 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9225d75c-1d9e-45e0-a415-34d8b60be54b-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:55.762209 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.762103 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" Oct 31 03:53:55.762209 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.762175 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph" event={"ID":"9225d75c-1d9e-45e0-a415-34d8b60be54b","Type":"ContainerDied","Data":"c64efca5d70e0a71e989a6067425a8f7db0649be45cea1442095afec11cb338e"} Oct 31 03:53:55.762697 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.762234 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64efca5d70e0a71e989a6067425a8f7db0649be45cea1442095afec11cb338e" Oct 31 03:53:55.764386 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.764339 4426 generic.go:356] "Generic (PLEG): container finished" podID="c98d149f-1eec-4e20-bcd7-2a4d7a89bb12" containerID="82dfae5a11a1a0ef54689eece6e386dd5f5f6b440af2b24c7db2a6e2fa587137" exitCode=0 Oct 31 03:53:55.764495 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:55.764419 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" event={"ID":"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12","Type":"ContainerDied","Data":"82dfae5a11a1a0ef54689eece6e386dd5f5f6b440af2b24c7db2a6e2fa587137"} Oct 31 03:53:56.937459 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:56.937430 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:57.063002 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.062915 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-xks-ca\") pod \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " Oct 31 03:53:57.063205 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.063035 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-kube-api-access-4v497\" (UniqueName: \"kubernetes.io/projected/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-kube-api-access-4v497\") pod \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " Oct 31 03:53:57.063588 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.063293 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-openshift-ca\") pod \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\" (UID: \"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12\") " Oct 31 03:53:57.063823 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.063797 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-openshift-ca" pod "c98d149f-1eec-4e20-bcd7-2a4d7a89bb12" (UID: "c98d149f-1eec-4e20-bcd7-2a4d7a89bb12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:57.070395 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.070304 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-xks-ca" pod "c98d149f-1eec-4e20-bcd7-2a4d7a89bb12" (UID: "c98d149f-1eec-4e20-bcd7-2a4d7a89bb12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:57.070395 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.070358 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-kube-api-access-4v497" pod "c98d149f-1eec-4e20-bcd7-2a4d7a89bb12" (UID: "c98d149f-1eec-4e20-bcd7-2a4d7a89bb12"). InnerVolumeSpecName "kube-api-access-4v497". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:57.164864 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.164829 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:57.164864 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.164862 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:57.164864 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.164872 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v497\" (UniqueName: \"kubernetes.io/projected/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12-kube-api-access-4v497\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:57.437231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.437188 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7"] Oct 31 03:53:57.438220 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.438198 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c98d149f-1eec-4e20-bcd7-2a4d7a89bb12" containerName="revoke-keys" Oct 31 03:53:57.438290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.438223 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98d149f-1eec-4e20-bcd7-2a4d7a89bb12" containerName="revoke-keys" Oct 31 03:53:57.438290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.438282 4426 cpu_manager.go:414] "containerMap: removing container" podUID="9225d75c-1d9e-45e0-a415-34d8b60be54b" containerName="revoke-keys" Oct 31 03:53:57.438356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.438290 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9225d75c-1d9e-45e0-a415-34d8b60be54b" containerName="revoke-keys" Oct 31 03:53:57.438450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.438438 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98d149f-1eec-4e20-bcd7-2a4d7a89bb12" containerName="revoke-keys" Oct 31 03:53:57.438486 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.438462 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="9225d75c-1d9e-45e0-a415-34d8b60be54b" containerName="revoke-keys" Oct 31 03:53:57.443066 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.443042 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:53:57.494438 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.494407 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7"] Oct 31 03:53:57.571463 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.571420 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dh2t\" (UniqueName: \"kubernetes.io/projected/0a75e294-8bd3-4629-9ce9-45d4203215a2-kube-api-access-4dh2t\") pod \"maas-api-revoke-sub-69f73a7b-zwtg7\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:53:57.571645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.571486 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69f73a7b-zwtg7\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:53:57.571645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.571623 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69f73a7b-zwtg7\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:53:57.673112 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.673065 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dh2t\" (UniqueName: \"kubernetes.io/projected/0a75e294-8bd3-4629-9ce9-45d4203215a2-kube-api-access-4dh2t\") pod \"maas-api-revoke-sub-69f73a7b-zwtg7\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:53:57.673297 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.673166 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69f73a7b-zwtg7\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:53:57.673297 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.673265 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69f73a7b-zwtg7\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:53:57.673980 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.673956 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69f73a7b-zwtg7\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:53:57.675623 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.675594 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69f73a7b-zwtg7\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:53:57.739110 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.739044 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dh2t\" (UniqueName: \"kubernetes.io/projected/0a75e294-8bd3-4629-9ce9-45d4203215a2-kube-api-access-4dh2t\") pod \"maas-api-revoke-sub-69f73a7b-zwtg7\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:53:57.758946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.758912 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:53:57.784919 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.784893 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" Oct 31 03:53:57.785046 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.784915 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw" event={"ID":"c98d149f-1eec-4e20-bcd7-2a4d7a89bb12","Type":"ContainerDied","Data":"cc17060e62a5bf3897b1f5acd3df0896b5cd003a0670261e37c51a1fd9bdc8bb"} Oct 31 03:53:57.785046 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:57.784984 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc17060e62a5bf3897b1f5acd3df0896b5cd003a0670261e37c51a1fd9bdc8bb" Oct 31 03:53:58.185329 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:58.185191 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7"] Oct 31 03:53:58.188298 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:53:58.188266 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a75e294_8bd3_4629_9ce9_45d4203215a2.slice/crio-67df52f487dfbc11cdfce2b0af3dc6a86247da605272bbcc21a51675d34ff965 WatchSource:0}: Error finding container 67df52f487dfbc11cdfce2b0af3dc6a86247da605272bbcc21a51675d34ff965: Status 404 returned error can't find the container with id 67df52f487dfbc11cdfce2b0af3dc6a86247da605272bbcc21a51675d34ff965 Oct 31 03:53:58.795973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:58.795871 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" event={"ID":"0a75e294-8bd3-4629-9ce9-45d4203215a2","Type":"ContainerStarted","Data":"f623977886c9c6fbaf7f3f52346c2c602745a65e02e2517a98aaa6962d56d07e"} Oct 31 03:53:58.795973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:58.795929 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" event={"ID":"0a75e294-8bd3-4629-9ce9-45d4203215a2","Type":"ContainerStarted","Data":"67df52f487dfbc11cdfce2b0af3dc6a86247da605272bbcc21a51675d34ff965"} Oct 31 03:53:58.906941 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:58.906887 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" podStartSLOduration=1.906870885 podStartE2EDuration="1.906870885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:58.904006444 +0000 UTC m=+1535.904631678" watchObservedRunningTime="2026-10-31 03:53:58.906870885 +0000 UTC m=+1535.907496135" Oct 31 03:53:59.807021 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:59.806978 4426 generic.go:356] "Generic (PLEG): container finished" podID="0a75e294-8bd3-4629-9ce9-45d4203215a2" containerID="f623977886c9c6fbaf7f3f52346c2c602745a65e02e2517a98aaa6962d56d07e" exitCode=0 Oct 31 03:53:59.807500 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:53:59.807068 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" event={"ID":"0a75e294-8bd3-4629-9ce9-45d4203215a2","Type":"ContainerDied","Data":"f623977886c9c6fbaf7f3f52346c2c602745a65e02e2517a98aaa6962d56d07e"} Oct 31 03:54:00.997526 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:00.997494 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:54:01.124886 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.124734 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a75e294-8bd3-4629-9ce9-45d4203215a2-kube-api-access-4dh2t\" (UniqueName: \"kubernetes.io/projected/0a75e294-8bd3-4629-9ce9-45d4203215a2-kube-api-access-4dh2t\") pod \"0a75e294-8bd3-4629-9ce9-45d4203215a2\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " Oct 31 03:54:01.125070 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.124924 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-openshift-ca\") pod \"0a75e294-8bd3-4629-9ce9-45d4203215a2\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " Oct 31 03:54:01.125177 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.125153 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-xks-ca\") pod \"0a75e294-8bd3-4629-9ce9-45d4203215a2\" (UID: \"0a75e294-8bd3-4629-9ce9-45d4203215a2\") " Oct 31 03:54:01.128930 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.128901 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a75e294-8bd3-4629-9ce9-45d4203215a2-kube-api-access-4dh2t" pod "0a75e294-8bd3-4629-9ce9-45d4203215a2" (UID: "0a75e294-8bd3-4629-9ce9-45d4203215a2"). InnerVolumeSpecName "kube-api-access-4dh2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:01.136363 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.136305 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-openshift-ca" pod "0a75e294-8bd3-4629-9ce9-45d4203215a2" (UID: "0a75e294-8bd3-4629-9ce9-45d4203215a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:01.151632 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.151581 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-xks-ca" pod "0a75e294-8bd3-4629-9ce9-45d4203215a2" (UID: "0a75e294-8bd3-4629-9ce9-45d4203215a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:01.226866 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.226819 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:01.226866 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.226858 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a75e294-8bd3-4629-9ce9-45d4203215a2-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:01.226866 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.226869 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dh2t\" (UniqueName: \"kubernetes.io/projected/0a75e294-8bd3-4629-9ce9-45d4203215a2-kube-api-access-4dh2t\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:01.827702 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.827668 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" Oct 31 03:54:01.827702 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.827693 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7" event={"ID":"0a75e294-8bd3-4629-9ce9-45d4203215a2","Type":"ContainerDied","Data":"67df52f487dfbc11cdfce2b0af3dc6a86247da605272bbcc21a51675d34ff965"} Oct 31 03:54:01.827974 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:01.827744 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67df52f487dfbc11cdfce2b0af3dc6a86247da605272bbcc21a51675d34ff965" Oct 31 03:54:02.648614 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.648574 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8"] Oct 31 03:54:02.649261 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.649245 4426 cpu_manager.go:414] "containerMap: removing container" podUID="0a75e294-8bd3-4629-9ce9-45d4203215a2" containerName="revoke-keys" Oct 31 03:54:02.649261 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.649261 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a75e294-8bd3-4629-9ce9-45d4203215a2" containerName="revoke-keys" Oct 31 03:54:02.649391 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.649381 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a75e294-8bd3-4629-9ce9-45d4203215a2" containerName="revoke-keys" Oct 31 03:54:02.653799 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.653773 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:02.709142 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.709094 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8"] Oct 31 03:54:02.744636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.744586 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d9d4f8-72kr8\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:02.744801 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.744769 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d9d4f8-72kr8\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:02.745009 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.744991 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzrhz\" (UniqueName: \"kubernetes.io/projected/f2370901-ce84-4e56-a7aa-213c99b20923-kube-api-access-hzrhz\") pod \"maas-api-revoke-sub-63d9d4f8-72kr8\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:02.748780 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.748747 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:54:02.748933 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.748853 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:54:02.748933 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.748865 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:54:02.749117 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.749094 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:54:02.846586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.846551 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzrhz\" (UniqueName: \"kubernetes.io/projected/f2370901-ce84-4e56-a7aa-213c99b20923-kube-api-access-hzrhz\") pod \"maas-api-revoke-sub-63d9d4f8-72kr8\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:02.846747 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.846666 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d9d4f8-72kr8\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:02.846831 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.846795 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d9d4f8-72kr8\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:02.847223 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:54:02.847190 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:54:02.847662 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.847633 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d9d4f8-72kr8\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:02.849247 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.849224 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d9d4f8-72kr8\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:02.904467 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.904394 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzrhz\" (UniqueName: \"kubernetes.io/projected/f2370901-ce84-4e56-a7aa-213c99b20923-kube-api-access-hzrhz\") pod \"maas-api-revoke-sub-63d9d4f8-72kr8\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:02.968507 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:02.968472 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:03.352768 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:03.352601 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8"] Oct 31 03:54:03.356104 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:03.356070 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2370901_ce84_4e56_a7aa_213c99b20923.slice/crio-8630e016f7deee8f9180f77f01906b9ada7ee86ac34fb166be98a580f24088dd WatchSource:0}: Error finding container 8630e016f7deee8f9180f77f01906b9ada7ee86ac34fb166be98a580f24088dd: Status 404 returned error can't find the container with id 8630e016f7deee8f9180f77f01906b9ada7ee86ac34fb166be98a580f24088dd Oct 31 03:54:03.849206 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:03.849157 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" event={"ID":"f2370901-ce84-4e56-a7aa-213c99b20923","Type":"ContainerStarted","Data":"0ec6553fb396ab02d2c6994eb791bfa78af4af77d2b98ab9724a0636a25c1bab"} Oct 31 03:54:03.849667 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:03.849236 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" event={"ID":"f2370901-ce84-4e56-a7aa-213c99b20923","Type":"ContainerStarted","Data":"8630e016f7deee8f9180f77f01906b9ada7ee86ac34fb166be98a580f24088dd"} Oct 31 03:54:03.961189 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:03.961138 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" podStartSLOduration=1.961108447 podStartE2EDuration="1.961108447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:03.957361084 +0000 UTC m=+1540.957986311" watchObservedRunningTime="2026-10-31 03:54:03.961108447 +0000 UTC m=+1540.961733675" Oct 31 03:54:04.862119 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:04.862085 4426 generic.go:356] "Generic (PLEG): container finished" podID="f2370901-ce84-4e56-a7aa-213c99b20923" containerID="0ec6553fb396ab02d2c6994eb791bfa78af4af77d2b98ab9724a0636a25c1bab" exitCode=0 Oct 31 03:54:04.862562 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:04.862182 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" event={"ID":"f2370901-ce84-4e56-a7aa-213c99b20923","Type":"ContainerDied","Data":"0ec6553fb396ab02d2c6994eb791bfa78af4af77d2b98ab9724a0636a25c1bab"} Oct 31 03:54:05.808509 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:05.808469 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh"] Oct 31 03:54:05.812960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:05.812939 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:05.869022 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:05.868981 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh"] Oct 31 03:54:05.888315 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:05.888281 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a12f38-f6bsh\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:05.888581 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:05.888547 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a12f38-f6bsh\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:05.888712 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:05.888610 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8htb\" (UniqueName: \"kubernetes.io/projected/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-kube-api-access-v8htb\") pod \"maas-api-revoke-sub-a5a12f38-f6bsh\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:05.990504 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:05.990462 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a12f38-f6bsh\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:05.990708 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:05.990690 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a12f38-f6bsh\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:05.990792 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:05.990733 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8htb\" (UniqueName: \"kubernetes.io/projected/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-kube-api-access-v8htb\") pod \"maas-api-revoke-sub-a5a12f38-f6bsh\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:05.992060 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:05.992025 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a12f38-f6bsh\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:05.994916 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:05.994889 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a12f38-f6bsh\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:06.041468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.041432 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:06.048483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.048455 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8htb\" (UniqueName: \"kubernetes.io/projected/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-kube-api-access-v8htb\") pod \"maas-api-revoke-sub-a5a12f38-f6bsh\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:06.091345 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.091264 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-xks-ca\") pod \"f2370901-ce84-4e56-a7aa-213c99b20923\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " Oct 31 03:54:06.091345 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.091310 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2370901-ce84-4e56-a7aa-213c99b20923-kube-api-access-hzrhz\" (UniqueName: \"kubernetes.io/projected/f2370901-ce84-4e56-a7aa-213c99b20923-kube-api-access-hzrhz\") pod \"f2370901-ce84-4e56-a7aa-213c99b20923\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " Oct 31 03:54:06.091345 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.091340 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-openshift-ca\") pod \"f2370901-ce84-4e56-a7aa-213c99b20923\" (UID: \"f2370901-ce84-4e56-a7aa-213c99b20923\") " Oct 31 03:54:06.091772 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.091749 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-openshift-ca" pod "f2370901-ce84-4e56-a7aa-213c99b20923" (UID: "f2370901-ce84-4e56-a7aa-213c99b20923"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:06.091947 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.091932 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:06.093441 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.093418 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-xks-ca" pod "f2370901-ce84-4e56-a7aa-213c99b20923" (UID: "f2370901-ce84-4e56-a7aa-213c99b20923"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:06.093524 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.093462 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2370901-ce84-4e56-a7aa-213c99b20923-kube-api-access-hzrhz" pod "f2370901-ce84-4e56-a7aa-213c99b20923" (UID: "f2370901-ce84-4e56-a7aa-213c99b20923"). InnerVolumeSpecName "kube-api-access-hzrhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:06.127717 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.127678 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:06.193439 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.193401 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2370901-ce84-4e56-a7aa-213c99b20923-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:06.193630 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.193616 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzrhz\" (UniqueName: \"kubernetes.io/projected/f2370901-ce84-4e56-a7aa-213c99b20923-kube-api-access-hzrhz\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:06.529942 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:06.529902 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27560d7d_ccf1_4bf1_a051_bfa1f0666e3a.slice/crio-49669f73f545cfb2db0fc8c6c5ce8192d2475ef5ebc7f204f99bf140524b342d WatchSource:0}: Error finding container 49669f73f545cfb2db0fc8c6c5ce8192d2475ef5ebc7f204f99bf140524b342d: Status 404 returned error can't find the container with id 49669f73f545cfb2db0fc8c6c5ce8192d2475ef5ebc7f204f99bf140524b342d Oct 31 03:54:06.529942 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.529916 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh"] Oct 31 03:54:06.885598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.885566 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" Oct 31 03:54:06.885598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.885584 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8" event={"ID":"f2370901-ce84-4e56-a7aa-213c99b20923","Type":"ContainerDied","Data":"8630e016f7deee8f9180f77f01906b9ada7ee86ac34fb166be98a580f24088dd"} Oct 31 03:54:06.886053 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.885639 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8630e016f7deee8f9180f77f01906b9ada7ee86ac34fb166be98a580f24088dd" Oct 31 03:54:06.887635 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.887612 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" event={"ID":"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a","Type":"ContainerStarted","Data":"8bdc2507ebc7337ba2277fab99f60647848a782361454a3458681df6e0dfe6bb"} Oct 31 03:54:06.887742 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:06.887661 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" event={"ID":"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a","Type":"ContainerStarted","Data":"49669f73f545cfb2db0fc8c6c5ce8192d2475ef5ebc7f204f99bf140524b342d"} Oct 31 03:54:07.002652 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:07.002590 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" podStartSLOduration=2.002574546 podStartE2EDuration="2.002574546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:06.998403279 +0000 UTC m=+1543.999028506" watchObservedRunningTime="2026-10-31 03:54:07.002574546 +0000 UTC m=+1544.003199835" Oct 31 03:54:07.900254 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:07.900215 4426 generic.go:356] "Generic (PLEG): container finished" podID="27560d7d-ccf1-4bf1-a051-bfa1f0666e3a" containerID="8bdc2507ebc7337ba2277fab99f60647848a782361454a3458681df6e0dfe6bb" exitCode=0 Oct 31 03:54:07.900692 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:07.900274 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" event={"ID":"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a","Type":"ContainerDied","Data":"8bdc2507ebc7337ba2277fab99f60647848a782361454a3458681df6e0dfe6bb"} Oct 31 03:54:08.184818 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.184726 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf"] Oct 31 03:54:08.185413 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.185397 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f2370901-ce84-4e56-a7aa-213c99b20923" containerName="revoke-keys" Oct 31 03:54:08.185454 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.185414 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2370901-ce84-4e56-a7aa-213c99b20923" containerName="revoke-keys" Oct 31 03:54:08.185539 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.185530 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2370901-ce84-4e56-a7aa-213c99b20923" containerName="revoke-keys" Oct 31 03:54:08.189100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.189078 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.250303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.250256 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf"] Oct 31 03:54:08.324968 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.324923 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-istio-token\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.325244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.325010 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-certs\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.325244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.325093 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.325244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.325168 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6brhl\" (UniqueName: \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-kube-api-access-6brhl\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.325244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.325208 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a303237c-f769-4565-bb71-de2d23a0519b-istiod-ca-cert\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.325244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.325240 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-credential-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.325518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.325302 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a303237c-f769-4565-bb71-de2d23a0519b-istio-podinfo\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.325518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.325369 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-data\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.325518 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.325439 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-envoy\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.427225 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.427171 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-data\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.427417 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.427240 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-envoy\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.427417 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.427326 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-istio-token\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.427417 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.427362 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-certs\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.427606 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.427430 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.427606 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.427486 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6brhl\" (UniqueName: \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-kube-api-access-6brhl\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.427606 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.427520 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a303237c-f769-4565-bb71-de2d23a0519b-istiod-ca-cert\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.427606 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.427552 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-credential-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.427805 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.427608 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a303237c-f769-4565-bb71-de2d23a0519b-istio-podinfo\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.427805 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.427611 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-data\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.428290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.427965 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-credential-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.428290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.428016 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-certs\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.428290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.428228 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-socket\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.428554 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.428533 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a303237c-f769-4565-bb71-de2d23a0519b-istiod-ca-cert\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.430082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.430052 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-envoy\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.430237 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.430146 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a303237c-f769-4565-bb71-de2d23a0519b-istio-podinfo\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.485086 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.485008 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-istio-token\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.485248 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.485185 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6brhl\" (UniqueName: \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-kube-api-access-6brhl\") pod \"e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.504260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.504220 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:08.954500 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.954467 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf"] Oct 31 03:54:08.959508 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:08.959478 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda303237c_f769_4565_bb71_de2d23a0519b.slice/crio-690686a732f0238f5729ed44c64698ce50abf26822a7f1390111749e46b5ab99 WatchSource:0}: Error finding container 690686a732f0238f5729ed44c64698ce50abf26822a7f1390111749e46b5ab99: Status 404 returned error can't find the container with id 690686a732f0238f5729ed44c64698ce50abf26822a7f1390111749e46b5ab99 Oct 31 03:54:08.962052 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.962005 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:08.962186 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.962103 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:08.962186 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:08.962174 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:09.078251 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.078204 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:09.138645 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.138604 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-kube-api-access-v8htb\" (UniqueName: \"kubernetes.io/projected/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-kube-api-access-v8htb\") pod \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " Oct 31 03:54:09.138832 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.138704 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-xks-ca\") pod \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " Oct 31 03:54:09.138832 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.138746 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-openshift-ca\") pod \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\" (UID: \"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a\") " Oct 31 03:54:09.139523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.139490 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-openshift-ca" pod "27560d7d-ccf1-4bf1-a051-bfa1f0666e3a" (UID: "27560d7d-ccf1-4bf1-a051-bfa1f0666e3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:09.141723 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.141689 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-xks-ca" pod "27560d7d-ccf1-4bf1-a051-bfa1f0666e3a" (UID: "27560d7d-ccf1-4bf1-a051-bfa1f0666e3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:09.142408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.142383 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-kube-api-access-v8htb" pod "27560d7d-ccf1-4bf1-a051-bfa1f0666e3a" (UID: "27560d7d-ccf1-4bf1-a051-bfa1f0666e3a"). InnerVolumeSpecName "kube-api-access-v8htb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:09.245019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.244924 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8htb\" (UniqueName: \"kubernetes.io/projected/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-kube-api-access-v8htb\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:09.245245 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.245223 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:09.246014 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.245349 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:09.963146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.962893 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" event={"ID":"a303237c-f769-4565-bb71-de2d23a0519b","Type":"ContainerStarted","Data":"755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226"} Oct 31 03:54:09.963146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.962977 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" event={"ID":"a303237c-f769-4565-bb71-de2d23a0519b","Type":"ContainerStarted","Data":"690686a732f0238f5729ed44c64698ce50abf26822a7f1390111749e46b5ab99"} Oct 31 03:54:09.966704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.966673 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" event={"ID":"27560d7d-ccf1-4bf1-a051-bfa1f0666e3a","Type":"ContainerDied","Data":"49669f73f545cfb2db0fc8c6c5ce8192d2475ef5ebc7f204f99bf140524b342d"} Oct 31 03:54:09.966906 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.966894 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49669f73f545cfb2db0fc8c6c5ce8192d2475ef5ebc7f204f99bf140524b342d" Oct 31 03:54:09.967017 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:09.967008 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh" Oct 31 03:54:10.130148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:10.125393 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" podStartSLOduration=2.125374048 podStartE2EDuration="2.125374048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:10.122918702 +0000 UTC m=+1547.123543950" watchObservedRunningTime="2026-10-31 03:54:10.125374048 +0000 UTC m=+1547.125999277" Oct 31 03:54:10.504733 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:10.504694 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:10.506344 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:10.506319 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:54:10.506454 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:10.506383 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" podUID="a303237c-f769-4565-bb71-de2d23a0519b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:54:11.053815 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:11.053781 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7"] Oct 31 03:54:11.097346 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:11.097310 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc8e788-nbzr7"] Oct 31 03:54:11.158683 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:11.158639 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="196fbf6e-b1f8-42ea-938f-2f1cde739855" path="/var/lib/kubelet/pods/196fbf6e-b1f8-42ea-938f-2f1cde739855/volumes" Oct 31 03:54:11.504984 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:11.504749 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:54:11.504984 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:11.504821 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" podUID="a303237c-f769-4565-bb71-de2d23a0519b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:54:11.941148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:11.940620 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m"] Oct 31 03:54:11.950151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:11.941700 4426 cpu_manager.go:414] "containerMap: removing container" podUID="27560d7d-ccf1-4bf1-a051-bfa1f0666e3a" containerName="revoke-keys" Oct 31 03:54:11.950151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:11.941725 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27560d7d-ccf1-4bf1-a051-bfa1f0666e3a" containerName="revoke-keys" Oct 31 03:54:11.950151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:11.941886 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="27560d7d-ccf1-4bf1-a051-bfa1f0666e3a" containerName="revoke-keys" Oct 31 03:54:11.956721 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:11.955529 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:11.978420 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:11.978389 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m"] Oct 31 03:54:12.030773 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.029405 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:54:12.030773 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.029460 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:54:12.030773 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.029504 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:54:12.030773 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.030167 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:54:12.096315 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.096284 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba5953b-322e-46f3-913e-637454ea624a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17869026-g745m\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:12.096691 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.096587 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba5953b-322e-46f3-913e-637454ea624a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17869026-g745m\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:12.096691 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.096661 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxsmz\" (UniqueName: \"kubernetes.io/projected/dba5953b-322e-46f3-913e-637454ea624a-kube-api-access-wxsmz\") pod \"maas-api-revoke-sub-17869026-g745m\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:12.198195 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.198070 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba5953b-322e-46f3-913e-637454ea624a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17869026-g745m\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:12.198195 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.198166 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsmz\" (UniqueName: \"kubernetes.io/projected/dba5953b-322e-46f3-913e-637454ea624a-kube-api-access-wxsmz\") pod \"maas-api-revoke-sub-17869026-g745m\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:12.198385 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.198213 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba5953b-322e-46f3-913e-637454ea624a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17869026-g745m\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:12.198862 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.198844 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba5953b-322e-46f3-913e-637454ea624a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17869026-g745m\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:12.200754 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.200731 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba5953b-322e-46f3-913e-637454ea624a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17869026-g745m\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:12.257423 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.257388 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxsmz\" (UniqueName: \"kubernetes.io/projected/dba5953b-322e-46f3-913e-637454ea624a-kube-api-access-wxsmz\") pod \"maas-api-revoke-sub-17869026-g745m\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:12.276071 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.276040 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:12.505294 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.505200 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:54:12.505528 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.505283 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" podUID="a303237c-f769-4565-bb71-de2d23a0519b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:54:12.703369 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.703338 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m"] Oct 31 03:54:12.706297 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:12.706264 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba5953b_322e_46f3_913e_637454ea624a.slice/crio-23bb0c98e687dca24dbc23286e5ad1629a66c715794d9de5f652c77c3e3ac07b WatchSource:0}: Error finding container 23bb0c98e687dca24dbc23286e5ad1629a66c715794d9de5f652c77c3e3ac07b: Status 404 returned error can't find the container with id 23bb0c98e687dca24dbc23286e5ad1629a66c715794d9de5f652c77c3e3ac07b Oct 31 03:54:12.998305 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:12.998267 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" event={"ID":"dba5953b-322e-46f3-913e-637454ea624a","Type":"ContainerStarted","Data":"23bb0c98e687dca24dbc23286e5ad1629a66c715794d9de5f652c77c3e3ac07b"} Oct 31 03:54:13.505479 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:13.505390 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:54:13.505479 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:13.505446 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" podUID="a303237c-f769-4565-bb71-de2d23a0519b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:54:14.009197 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:14.009159 4426 generic.go:356] "Generic (PLEG): container finished" podID="dba5953b-322e-46f3-913e-637454ea624a" containerID="d2c2c3a01f5f24feb886403a79703797d995fb3637de8e229e683cd08eb6cb8c" exitCode=0 Oct 31 03:54:14.009380 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:14.009242 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" event={"ID":"dba5953b-322e-46f3-913e-637454ea624a","Type":"ContainerDied","Data":"d2c2c3a01f5f24feb886403a79703797d995fb3637de8e229e683cd08eb6cb8c"} Oct 31 03:54:14.505091 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:14.505056 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:54:14.505302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:14.505152 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" podUID="a303237c-f769-4565-bb71-de2d23a0519b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:54:15.293501 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.293469 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:15.481112 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.481071 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba5953b-322e-46f3-913e-637454ea624a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba5953b-322e-46f3-913e-637454ea624a-maas-api-openshift-ca\") pod \"dba5953b-322e-46f3-913e-637454ea624a\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " Oct 31 03:54:15.481338 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.481187 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dba5953b-322e-46f3-913e-637454ea624a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba5953b-322e-46f3-913e-637454ea624a-maas-api-xks-ca\") pod \"dba5953b-322e-46f3-913e-637454ea624a\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " Oct 31 03:54:15.481482 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.481461 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba5953b-322e-46f3-913e-637454ea624a-kube-api-access-wxsmz\" (UniqueName: \"kubernetes.io/projected/dba5953b-322e-46f3-913e-637454ea624a-kube-api-access-wxsmz\") pod \"dba5953b-322e-46f3-913e-637454ea624a\" (UID: \"dba5953b-322e-46f3-913e-637454ea624a\") " Oct 31 03:54:15.481536 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.481507 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba5953b-322e-46f3-913e-637454ea624a-maas-api-openshift-ca" pod "dba5953b-322e-46f3-913e-637454ea624a" (UID: "dba5953b-322e-46f3-913e-637454ea624a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:15.482052 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.482027 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba5953b-322e-46f3-913e-637454ea624a-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:15.483287 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.483253 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba5953b-322e-46f3-913e-637454ea624a-maas-api-xks-ca" pod "dba5953b-322e-46f3-913e-637454ea624a" (UID: "dba5953b-322e-46f3-913e-637454ea624a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:15.483465 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.483447 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba5953b-322e-46f3-913e-637454ea624a-kube-api-access-wxsmz" pod "dba5953b-322e-46f3-913e-637454ea624a" (UID: "dba5953b-322e-46f3-913e-637454ea624a"). InnerVolumeSpecName "kube-api-access-wxsmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:15.505446 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.505407 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:54:15.505579 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.505479 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" podUID="a303237c-f769-4565-bb71-de2d23a0519b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:54:15.583111 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.583072 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba5953b-322e-46f3-913e-637454ea624a-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:15.583111 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.583105 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxsmz\" (UniqueName: \"kubernetes.io/projected/dba5953b-322e-46f3-913e-637454ea624a-kube-api-access-wxsmz\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:15.978613 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:15.978574 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:54:16.068427 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.068385 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" event={"ID":"dba5953b-322e-46f3-913e-637454ea624a","Type":"ContainerDied","Data":"23bb0c98e687dca24dbc23286e5ad1629a66c715794d9de5f652c77c3e3ac07b"} Oct 31 03:54:16.068595 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.068454 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23bb0c98e687dca24dbc23286e5ad1629a66c715794d9de5f652c77c3e3ac07b" Oct 31 03:54:16.068595 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.068403 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m" Oct 31 03:54:16.706545 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.706510 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:16.708356 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.708337 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:16.714493 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.714471 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 03:54:16.872252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.872212 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx"] Oct 31 03:54:16.873298 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.873271 4426 cpu_manager.go:414] "containerMap: removing container" podUID="dba5953b-322e-46f3-913e-637454ea624a" containerName="revoke-keys" Oct 31 03:54:16.873298 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.873297 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba5953b-322e-46f3-913e-637454ea624a" containerName="revoke-keys" Oct 31 03:54:16.873498 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.873484 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba5953b-322e-46f3-913e-637454ea624a" containerName="revoke-keys" Oct 31 03:54:16.879153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.878252 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:16.937983 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.937946 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:54:16.938184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.938096 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:54:16.939005 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.938982 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:54:16.967227 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.939265 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:54:16.967483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:16.940028 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx"] Oct 31 03:54:17.014464 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.014424 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a85fb4fb-dcmvx\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:17.014815 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.014793 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a85fb4fb-dcmvx\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:17.015096 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.015076 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crq8w\" (UniqueName: \"kubernetes.io/projected/41025c7d-8f03-4f3e-9926-aa3408bf86de-kube-api-access-crq8w\") pod \"maas-api-revoke-sub-a85fb4fb-dcmvx\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:17.079231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.079194 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp"] Oct 31 03:54:17.084623 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.084594 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.128155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.116719 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crq8w\" (UniqueName: \"kubernetes.io/projected/41025c7d-8f03-4f3e-9926-aa3408bf86de-kube-api-access-crq8w\") pod \"maas-api-revoke-sub-a85fb4fb-dcmvx\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:17.128155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.117185 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a85fb4fb-dcmvx\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:17.128155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.117241 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a85fb4fb-dcmvx\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:17.128155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.119296 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a85fb4fb-dcmvx\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:17.128155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.126570 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a85fb4fb-dcmvx\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:17.241890 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.241794 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp"] Oct 31 03:54:17.266098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.263925 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-data\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.266098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.263988 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0a72fe-ce05-498d-8e57-bb41826904a0-istiod-ca-cert\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.266098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.264045 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-podinfo\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.266098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.264095 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-token\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.266098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.264285 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-certs\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.266098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.264327 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-envoy\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.266098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.264434 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.266098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.264471 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-credential-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.266098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.264522 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4t4h\" (UniqueName: \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-kube-api-access-r4t4h\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.271718 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.271686 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crq8w\" (UniqueName: \"kubernetes.io/projected/41025c7d-8f03-4f3e-9926-aa3408bf86de-kube-api-access-crq8w\") pod \"maas-api-revoke-sub-a85fb4fb-dcmvx\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.379698 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.379761 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-credential-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.379801 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4t4h\" (UniqueName: \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-kube-api-access-r4t4h\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.379864 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-data\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.379912 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0a72fe-ce05-498d-8e57-bb41826904a0-istiod-ca-cert\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.379973 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-podinfo\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.380028 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-token\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.380186 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-certs\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.380241 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-envoy\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.382031 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-data\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.382499 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.383146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.382771 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-credential-socket\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.390047 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.384449 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0a72fe-ce05-498d-8e57-bb41826904a0-istiod-ca-cert\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.390047 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.384725 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-certs\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.408437 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.398525 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-envoy\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.408437 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.405855 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-podinfo\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.454150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.446324 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4t4h\" (UniqueName: \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-kube-api-access-r4t4h\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.454150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.453752 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-token\") pod \"e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:17.523710 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.523359 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:17.786044 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:17.786013 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:18.534475 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:18.525261 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx"] Oct 31 03:54:19.083159 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:19.082173 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:19.312220 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:19.311375 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:19.477483 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:19.477429 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Oct 31 03:54:19.490415 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:19.490350 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 31 03:54:19.521164 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:19.518260 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:19.522755 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:19.522702 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:19.613210 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:19.612328 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:19.754675 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:19.754617 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:20.004193 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:19.977431 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:20.004193 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:19.978575 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:20.150151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.125608 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:20.206159 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.204369 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp"] Oct 31 03:54:20.235152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.221527 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Oct 31 03:54:20.261712 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.247640 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:20.261712 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.248487 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:54:20.316148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.315450 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 03:54:20.316148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.315884 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:54:20.607351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.607250 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:20.625776 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.625734 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Oct 31 03:54:20.656278 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.626057 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:54:20.656590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.656567 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:54:20.660688 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.658744 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:54:20.660688 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.659163 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:54:20.703523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.703460 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 03:54:20.704269 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.704088 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:54:20.708807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.708608 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:20.708807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.708711 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:20.708807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.708752 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:20.733294 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.733267 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:54:20.747285 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.745975 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:54:20.747285 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.746907 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:54:20.754998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.751012 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:54:20.754998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.752563 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" event={"ID":"41025c7d-8f03-4f3e-9926-aa3408bf86de","Type":"ContainerStarted","Data":"19398a9478ebf27c2ad306ba152231fb517747b0cbea6957c38c0b80d6be736d"} Oct 31 03:54:20.754998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.752616 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" event={"ID":"41025c7d-8f03-4f3e-9926-aa3408bf86de","Type":"ContainerStarted","Data":"2eb371f8c75342a46bc0f93e7c0ca9197e5f64a16423acbb320a3da1bf8cc4ca"} Oct 31 03:54:20.766848 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.765509 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:54:20.774166 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:20.771059 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:54:20.808018 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:54:20.807956 4426 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41025c7d_8f03_4f3e_9926_aa3408bf86de.slice/crio-conmon-19398a9478ebf27c2ad306ba152231fb517747b0cbea6957c38c0b80d6be736d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41025c7d_8f03_4f3e_9926_aa3408bf86de.slice/crio-19398a9478ebf27c2ad306ba152231fb517747b0cbea6957c38c0b80d6be736d.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:54:21.777827 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.777729 4426 generic.go:356] "Generic (PLEG): container finished" podID="41025c7d-8f03-4f3e-9926-aa3408bf86de" containerID="19398a9478ebf27c2ad306ba152231fb517747b0cbea6957c38c0b80d6be736d" exitCode=0 Oct 31 03:54:21.778800 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.778087 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" event={"ID":"41025c7d-8f03-4f3e-9926-aa3408bf86de","Type":"ContainerDied","Data":"19398a9478ebf27c2ad306ba152231fb517747b0cbea6957c38c0b80d6be736d"} Oct 31 03:54:21.785729 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.785688 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" event={"ID":"7f0a72fe-ce05-498d-8e57-bb41826904a0","Type":"ContainerStarted","Data":"0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db"} Oct 31 03:54:21.785843 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.785749 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" event={"ID":"7f0a72fe-ce05-498d-8e57-bb41826904a0","Type":"ContainerStarted","Data":"59ab2338d8176dc9be7d3472dabce2cb661846778973686b571429c9e155e983"} Oct 31 03:54:21.791158 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.790098 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:21.791158 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.790211 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:54:21.791158 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.790285 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:54:21.865145 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.864183 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx"] Oct 31 03:54:21.882162 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.881516 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:21.911974 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.909695 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx"] Oct 31 03:54:21.976091 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.976051 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c61a59f-mx2gx\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:21.976501 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.976480 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c61a59f-mx2gx\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:21.976703 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:21.976687 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrkjc\" (UniqueName: \"kubernetes.io/projected/cc366446-32f5-413f-896f-58b60f082b5e-kube-api-access-qrkjc\") pod \"maas-api-revoke-sub-3c61a59f-mx2gx\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:22.078335 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.078305 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c61a59f-mx2gx\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:22.078458 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.078402 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrkjc\" (UniqueName: \"kubernetes.io/projected/cc366446-32f5-413f-896f-58b60f082b5e-kube-api-access-qrkjc\") pod \"maas-api-revoke-sub-3c61a59f-mx2gx\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:22.078532 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.078516 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c61a59f-mx2gx\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:22.079199 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.079179 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c61a59f-mx2gx\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:22.080923 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.080900 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c61a59f-mx2gx\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:22.103979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.103947 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:22.155732 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.155663 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" podStartSLOduration=5.1556405 podStartE2EDuration="5.1556405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:22.139205706 +0000 UTC m=+1559.139830935" watchObservedRunningTime="2026-10-31 03:54:22.1556405 +0000 UTC m=+1559.156265729" Oct 31 03:54:22.156630 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.156603 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrkjc\" (UniqueName: \"kubernetes.io/projected/cc366446-32f5-413f-896f-58b60f082b5e-kube-api-access-qrkjc\") pod \"maas-api-revoke-sub-3c61a59f-mx2gx\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:22.179485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.179445 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-xks-ca\") pod \"41025c7d-8f03-4f3e-9926-aa3408bf86de\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " Oct 31 03:54:22.179683 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.179655 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41025c7d-8f03-4f3e-9926-aa3408bf86de-kube-api-access-crq8w\" (UniqueName: \"kubernetes.io/projected/41025c7d-8f03-4f3e-9926-aa3408bf86de-kube-api-access-crq8w\") pod \"41025c7d-8f03-4f3e-9926-aa3408bf86de\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " Oct 31 03:54:22.179802 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.179788 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-openshift-ca\") pod \"41025c7d-8f03-4f3e-9926-aa3408bf86de\" (UID: \"41025c7d-8f03-4f3e-9926-aa3408bf86de\") " Oct 31 03:54:22.180176 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.180144 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-openshift-ca" pod "41025c7d-8f03-4f3e-9926-aa3408bf86de" (UID: "41025c7d-8f03-4f3e-9926-aa3408bf86de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:22.181727 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.181696 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41025c7d-8f03-4f3e-9926-aa3408bf86de-kube-api-access-crq8w" pod "41025c7d-8f03-4f3e-9926-aa3408bf86de" (UID: "41025c7d-8f03-4f3e-9926-aa3408bf86de"). InnerVolumeSpecName "kube-api-access-crq8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:22.181727 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.181702 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-xks-ca" pod "41025c7d-8f03-4f3e-9926-aa3408bf86de" (UID: "41025c7d-8f03-4f3e-9926-aa3408bf86de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:22.203026 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.202982 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:22.281991 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.281945 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:22.281991 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.281980 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crq8w\" (UniqueName: \"kubernetes.io/projected/41025c7d-8f03-4f3e-9926-aa3408bf86de-kube-api-access-crq8w\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:22.281991 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.281992 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41025c7d-8f03-4f3e-9926-aa3408bf86de-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:22.655082 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.655039 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx"] Oct 31 03:54:22.659585 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:22.659556 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc366446_32f5_413f_896f_58b60f082b5e.slice/crio-c6f5103baf2927f879cacba87197711fe4c6a392117169a8da77b93b3a05b7ef WatchSource:0}: Error finding container c6f5103baf2927f879cacba87197711fe4c6a392117169a8da77b93b3a05b7ef: Status 404 returned error can't find the container with id c6f5103baf2927f879cacba87197711fe4c6a392117169a8da77b93b3a05b7ef Oct 31 03:54:22.662244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.661758 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l"] Oct 31 03:54:22.662512 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.662497 4426 cpu_manager.go:414] "containerMap: removing container" podUID="41025c7d-8f03-4f3e-9926-aa3408bf86de" containerName="revoke-keys" Oct 31 03:54:22.662512 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.662511 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41025c7d-8f03-4f3e-9926-aa3408bf86de" containerName="revoke-keys" Oct 31 03:54:22.662656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.662639 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="41025c7d-8f03-4f3e-9926-aa3408bf86de" containerName="revoke-keys" Oct 31 03:54:22.668390 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.667705 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:22.687212 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.687180 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14769dc-mfg5l\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:22.687361 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.687340 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2gg6\" (UniqueName: \"kubernetes.io/projected/1397f512-8b19-4e94-881c-6030d5953ee7-kube-api-access-b2gg6\") pod \"maas-api-revoke-sub-f14769dc-mfg5l\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:22.687614 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.687593 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14769dc-mfg5l\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:22.774359 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.769811 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l"] Oct 31 03:54:22.788568 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.788533 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14769dc-mfg5l\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:22.788924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.788600 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14769dc-mfg5l\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:22.788924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.788660 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2gg6\" (UniqueName: \"kubernetes.io/projected/1397f512-8b19-4e94-881c-6030d5953ee7-kube-api-access-b2gg6\") pod \"maas-api-revoke-sub-f14769dc-mfg5l\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:22.789243 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.789142 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:54:22.789243 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.789193 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:54:22.789945 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.789916 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14769dc-mfg5l\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:22.796744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.796725 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" Oct 31 03:54:22.796982 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.796757 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx" event={"ID":"41025c7d-8f03-4f3e-9926-aa3408bf86de","Type":"ContainerDied","Data":"2eb371f8c75342a46bc0f93e7c0ca9197e5f64a16423acbb320a3da1bf8cc4ca"} Oct 31 03:54:22.797065 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.797006 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eb371f8c75342a46bc0f93e7c0ca9197e5f64a16423acbb320a3da1bf8cc4ca" Oct 31 03:54:22.798884 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.798855 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" event={"ID":"cc366446-32f5-413f-896f-58b60f082b5e","Type":"ContainerStarted","Data":"c6f5103baf2927f879cacba87197711fe4c6a392117169a8da77b93b3a05b7ef"} Oct 31 03:54:22.833140 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.833084 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14769dc-mfg5l\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:22.854400 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.854367 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2gg6\" (UniqueName: \"kubernetes.io/projected/1397f512-8b19-4e94-881c-6030d5953ee7-kube-api-access-b2gg6\") pod \"maas-api-revoke-sub-f14769dc-mfg5l\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:22.985031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:22.985000 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:23.212249 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.212150 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6965-15b1fc"] Oct 31 03:54:23.250341 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.250218 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-15b1fc" Oct 31 03:54:23.272267 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.271019 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6965-15b1fc"] Oct 31 03:54:23.297442 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.297406 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2kcn\" (UniqueName: \"kubernetes.io/projected/0523c1f2-0ee7-4452-b56e-92c4e792956e-kube-api-access-l2kcn\") pod \"test-curl-6965-15b1fc\" (UID: \"0523c1f2-0ee7-4452-b56e-92c4e792956e\") " pod="opendatahub/test-curl-6965-15b1fc" Oct 31 03:54:23.299423 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.299402 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2x2q5\"" Oct 31 03:54:23.399457 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.399422 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2kcn\" (UniqueName: \"kubernetes.io/projected/0523c1f2-0ee7-4452-b56e-92c4e792956e-kube-api-access-l2kcn\") pod \"test-curl-6965-15b1fc\" (UID: \"0523c1f2-0ee7-4452-b56e-92c4e792956e\") " pod="opendatahub/test-curl-6965-15b1fc" Oct 31 03:54:23.407809 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.407709 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l"] Oct 31 03:54:23.466860 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:23.411145 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1397f512_8b19_4e94_881c_6030d5953ee7.slice/crio-33c0d341c5225f118daddc8d5de7a09eeed14945ad8c811b479f9a5adaca8ec4 WatchSource:0}: Error finding container 33c0d341c5225f118daddc8d5de7a09eeed14945ad8c811b479f9a5adaca8ec4: Status 404 returned error can't find the container with id 33c0d341c5225f118daddc8d5de7a09eeed14945ad8c811b479f9a5adaca8ec4 Oct 31 03:54:23.466860 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.457331 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2kcn\" (UniqueName: \"kubernetes.io/projected/0523c1f2-0ee7-4452-b56e-92c4e792956e-kube-api-access-l2kcn\") pod \"test-curl-6965-15b1fc\" (UID: \"0523c1f2-0ee7-4452-b56e-92c4e792956e\") " pod="opendatahub/test-curl-6965-15b1fc" Oct 31 03:54:23.567963 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.567923 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-15b1fc" Oct 31 03:54:23.593555 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.593518 4426 scope.go:122] "RemoveContainer" containerID="9094672e56c4a453cc478580bf0bd2b2a59441f0f8acea0918d81499831d292b" Oct 31 03:54:23.604920 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.604894 4426 scope.go:122] "RemoveContainer" containerID="95a7ccd196bc168ea3389fd07aad680b91f97bbb529e2c55e8d146196026c17c" Oct 31 03:54:23.628151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.626119 4426 scope.go:122] "RemoveContainer" containerID="3e3c416b9496659c72ebe32983796a5feba43284cae37ff61bfea06a6d945baa" Oct 31 03:54:23.669583 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.669558 4426 scope.go:122] "RemoveContainer" containerID="04404620550be34553d1db7c941cecc2940e5296cf64015d0f29b959ec5a7c43" Oct 31 03:54:23.681257 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.681115 4426 scope.go:122] "RemoveContainer" containerID="5b3ff52feb81268cdbda876ad4f47e51df1d791676370b64f501f3b613a299c6" Oct 31 03:54:23.693346 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.693232 4426 scope.go:122] "RemoveContainer" containerID="fdb45fbdb638ff3630f03f4d1f39adcb6581ef97c80276e2472e3f14bf6df765" Oct 31 03:54:23.708176 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.708065 4426 scope.go:122] "RemoveContainer" containerID="4cb5ef8a3eb36d4a9dda21e61e1106d056f4394fa1dfc5159b4feb75794356f3" Oct 31 03:54:23.775868 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.775356 4426 scope.go:122] "RemoveContainer" containerID="e1fbd8925760e13d4c253d55578cafeeaca621c2356fad3bf3cdf736d58dffd9" Oct 31 03:54:23.791482 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.791442 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:54:23.792451 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.792416 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:54:23.868478 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.868237 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" event={"ID":"1397f512-8b19-4e94-881c-6030d5953ee7","Type":"ContainerStarted","Data":"33c0d341c5225f118daddc8d5de7a09eeed14945ad8c811b479f9a5adaca8ec4"} Oct 31 03:54:23.882093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.882065 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" event={"ID":"cc366446-32f5-413f-896f-58b60f082b5e","Type":"ContainerStarted","Data":"86eebec09fe3eaf108afb856ecc389a8d1011b875c4674fa38876ecff6402557"} Oct 31 03:54:23.998700 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:23.998593 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" podStartSLOduration=2.998577902 podStartE2EDuration="2.998577902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:23.99846095 +0000 UTC m=+1560.999086178" watchObservedRunningTime="2026-10-31 03:54:23.998577902 +0000 UTC m=+1560.999203130" Oct 31 03:54:24.068423 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:24.068256 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6965-15b1fc"] Oct 31 03:54:24.071237 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:24.071201 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0523c1f2_0ee7_4452_b56e_92c4e792956e.slice/crio-b474ce3fc818c769e6326bbf16d41a1aeb72f021f50107ef7fd27cf56fb77e36 WatchSource:0}: Error finding container b474ce3fc818c769e6326bbf16d41a1aeb72f021f50107ef7fd27cf56fb77e36: Status 404 returned error can't find the container with id b474ce3fc818c769e6326bbf16d41a1aeb72f021f50107ef7fd27cf56fb77e36 Oct 31 03:54:24.073261 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:24.073244 4426 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:54:24.789876 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:24.789786 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:54:24.789876 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:24.789849 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:54:24.892365 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:24.892327 4426 generic.go:356] "Generic (PLEG): container finished" podID="1397f512-8b19-4e94-881c-6030d5953ee7" containerID="d253d9e5fdc81611fe1d6e62302f522e4dcf0ae46c8d23deb29a574b457d6781" exitCode=0 Oct 31 03:54:24.892815 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:24.892417 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" event={"ID":"1397f512-8b19-4e94-881c-6030d5953ee7","Type":"ContainerDied","Data":"d253d9e5fdc81611fe1d6e62302f522e4dcf0ae46c8d23deb29a574b457d6781"} Oct 31 03:54:24.894312 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:24.894288 4426 generic.go:356] "Generic (PLEG): container finished" podID="cc366446-32f5-413f-896f-58b60f082b5e" containerID="86eebec09fe3eaf108afb856ecc389a8d1011b875c4674fa38876ecff6402557" exitCode=0 Oct 31 03:54:24.894452 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:24.894365 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" event={"ID":"cc366446-32f5-413f-896f-58b60f082b5e","Type":"ContainerDied","Data":"86eebec09fe3eaf108afb856ecc389a8d1011b875c4674fa38876ecff6402557"} Oct 31 03:54:24.895815 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:24.895795 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6965-15b1fc" event={"ID":"0523c1f2-0ee7-4452-b56e-92c4e792956e","Type":"ContainerStarted","Data":"b474ce3fc818c769e6326bbf16d41a1aeb72f021f50107ef7fd27cf56fb77e36"} Oct 31 03:54:25.789259 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:25.789223 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:54:25.789453 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:25.789274 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:54:25.906211 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:25.906175 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6965-15b1fc" event={"ID":"0523c1f2-0ee7-4452-b56e-92c4e792956e","Type":"ContainerStarted","Data":"30de8953bce89a966991c6379c93028f8aaf45dd3b80ce2b6c2ace078ff62f8c"} Oct 31 03:54:26.069199 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.067167 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6965-15b1fc" podStartSLOduration=1.7728779669999999 podStartE2EDuration="3.067144206s" totalImagesPullingTime="1.294266239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:26.059960578 +0000 UTC m=+1563.060585808" watchObservedRunningTime="2026-10-31 03:54:26.067144206 +0000 UTC m=+1563.067769435" Oct 31 03:54:26.078886 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.078836 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp"] Oct 31 03:54:26.084485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.084459 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:26.106503 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.106341 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp"] Oct 31 03:54:26.185639 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.184918 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab34e2db-vghsp\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:26.185639 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.185202 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab34e2db-vghsp\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:26.185639 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.185244 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ss8p\" (UniqueName: \"kubernetes.io/projected/8474f84c-56a3-4e73-8936-edd0986ab7ed-kube-api-access-9ss8p\") pod \"maas-api-revoke-sub-ab34e2db-vghsp\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:26.203025 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.202996 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:26.290470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.288668 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-openshift-ca\") pod \"1397f512-8b19-4e94-881c-6030d5953ee7\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " Oct 31 03:54:26.290470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.288763 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1397f512-8b19-4e94-881c-6030d5953ee7-kube-api-access-b2gg6\" (UniqueName: \"kubernetes.io/projected/1397f512-8b19-4e94-881c-6030d5953ee7-kube-api-access-b2gg6\") pod \"1397f512-8b19-4e94-881c-6030d5953ee7\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " Oct 31 03:54:26.290470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.288941 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-xks-ca\") pod \"1397f512-8b19-4e94-881c-6030d5953ee7\" (UID: \"1397f512-8b19-4e94-881c-6030d5953ee7\") " Oct 31 03:54:26.290470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.289282 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab34e2db-vghsp\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:26.290470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.289399 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-openshift-ca" pod "1397f512-8b19-4e94-881c-6030d5953ee7" (UID: "1397f512-8b19-4e94-881c-6030d5953ee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:26.290470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.289497 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab34e2db-vghsp\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:26.290470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.289541 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ss8p\" (UniqueName: \"kubernetes.io/projected/8474f84c-56a3-4e73-8936-edd0986ab7ed-kube-api-access-9ss8p\") pod \"maas-api-revoke-sub-ab34e2db-vghsp\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:26.290470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.289894 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:26.290470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.290403 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab34e2db-vghsp\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:26.291498 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.291473 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1397f512-8b19-4e94-881c-6030d5953ee7-kube-api-access-b2gg6" pod "1397f512-8b19-4e94-881c-6030d5953ee7" (UID: "1397f512-8b19-4e94-881c-6030d5953ee7"). InnerVolumeSpecName "kube-api-access-b2gg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:26.292198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.292172 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab34e2db-vghsp\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:26.292639 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.292608 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-xks-ca" pod "1397f512-8b19-4e94-881c-6030d5953ee7" (UID: "1397f512-8b19-4e94-881c-6030d5953ee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:26.298618 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.298596 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:26.366625 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.365429 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ss8p\" (UniqueName: \"kubernetes.io/projected/8474f84c-56a3-4e73-8936-edd0986ab7ed-kube-api-access-9ss8p\") pod \"maas-api-revoke-sub-ab34e2db-vghsp\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:26.391222 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.391192 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc366446-32f5-413f-896f-58b60f082b5e-kube-api-access-qrkjc\" (UniqueName: \"kubernetes.io/projected/cc366446-32f5-413f-896f-58b60f082b5e-kube-api-access-qrkjc\") pod \"cc366446-32f5-413f-896f-58b60f082b5e\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " Oct 31 03:54:26.391532 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.391518 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-xks-ca\") pod \"cc366446-32f5-413f-896f-58b60f082b5e\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " Oct 31 03:54:26.392029 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.392011 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-openshift-ca\") pod \"cc366446-32f5-413f-896f-58b60f082b5e\" (UID: \"cc366446-32f5-413f-896f-58b60f082b5e\") " Oct 31 03:54:26.392532 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.392502 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-openshift-ca" pod "cc366446-32f5-413f-896f-58b60f082b5e" (UID: "cc366446-32f5-413f-896f-58b60f082b5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:26.392840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.392826 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:26.392924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.392915 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1397f512-8b19-4e94-881c-6030d5953ee7-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:26.392982 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.392975 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2gg6\" (UniqueName: \"kubernetes.io/projected/1397f512-8b19-4e94-881c-6030d5953ee7-kube-api-access-b2gg6\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:26.393514 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.393490 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc366446-32f5-413f-896f-58b60f082b5e-kube-api-access-qrkjc" pod "cc366446-32f5-413f-896f-58b60f082b5e" (UID: "cc366446-32f5-413f-896f-58b60f082b5e"). InnerVolumeSpecName "kube-api-access-qrkjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:26.395132 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.395101 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-xks-ca" pod "cc366446-32f5-413f-896f-58b60f082b5e" (UID: "cc366446-32f5-413f-896f-58b60f082b5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:26.406027 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.406006 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:26.494264 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.494228 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrkjc\" (UniqueName: \"kubernetes.io/projected/cc366446-32f5-413f-896f-58b60f082b5e-kube-api-access-qrkjc\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:26.494264 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.494267 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc366446-32f5-413f-896f-58b60f082b5e-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:26.789571 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.789538 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:54:26.789757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.789593 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:54:26.878956 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.878930 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp"] Oct 31 03:54:26.881576 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:26.881547 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8474f84c_56a3_4e73_8936_edd0986ab7ed.slice/crio-abd8726d9633b71a041006f7964b8233eee517b8e0dc150df75725869a6ac9ea WatchSource:0}: Error finding container abd8726d9633b71a041006f7964b8233eee517b8e0dc150df75725869a6ac9ea: Status 404 returned error can't find the container with id abd8726d9633b71a041006f7964b8233eee517b8e0dc150df75725869a6ac9ea Oct 31 03:54:26.937149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.932514 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" event={"ID":"8474f84c-56a3-4e73-8936-edd0986ab7ed","Type":"ContainerStarted","Data":"abd8726d9633b71a041006f7964b8233eee517b8e0dc150df75725869a6ac9ea"} Oct 31 03:54:26.957144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.954635 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" Oct 31 03:54:26.957144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.954744 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx" event={"ID":"cc366446-32f5-413f-896f-58b60f082b5e","Type":"ContainerDied","Data":"c6f5103baf2927f879cacba87197711fe4c6a392117169a8da77b93b3a05b7ef"} Oct 31 03:54:26.957144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.954801 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6f5103baf2927f879cacba87197711fe4c6a392117169a8da77b93b3a05b7ef" Oct 31 03:54:26.970738 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.970575 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" Oct 31 03:54:26.970935 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.970768 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l" event={"ID":"1397f512-8b19-4e94-881c-6030d5953ee7","Type":"ContainerDied","Data":"33c0d341c5225f118daddc8d5de7a09eeed14945ad8c811b479f9a5adaca8ec4"} Oct 31 03:54:26.971727 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:26.971071 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33c0d341c5225f118daddc8d5de7a09eeed14945ad8c811b479f9a5adaca8ec4" Oct 31 03:54:27.600360 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.600320 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv"] Oct 31 03:54:27.602182 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.602100 4426 cpu_manager.go:414] "containerMap: removing container" podUID="cc366446-32f5-413f-896f-58b60f082b5e" containerName="revoke-keys" Oct 31 03:54:27.602341 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.602328 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc366446-32f5-413f-896f-58b60f082b5e" containerName="revoke-keys" Oct 31 03:54:27.602486 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.602475 4426 cpu_manager.go:414] "containerMap: removing container" podUID="1397f512-8b19-4e94-881c-6030d5953ee7" containerName="revoke-keys" Oct 31 03:54:27.602569 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.602559 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1397f512-8b19-4e94-881c-6030d5953ee7" containerName="revoke-keys" Oct 31 03:54:27.603095 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.603042 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc366446-32f5-413f-896f-58b60f082b5e" containerName="revoke-keys" Oct 31 03:54:27.603271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.603257 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="1397f512-8b19-4e94-881c-6030d5953ee7" containerName="revoke-keys" Oct 31 03:54:27.608998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.608969 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:27.619930 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.619892 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr"] Oct 31 03:54:27.625849 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.625820 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:27.660774 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.659256 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv"] Oct 31 03:54:27.684826 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.684789 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr"] Oct 31 03:54:27.712196 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.712161 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a17bdb58-rrcbv\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:27.712196 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.712198 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4284dcb-r6dsr\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:27.712427 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.712378 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdcqd\" (UniqueName: \"kubernetes.io/projected/311a2b24-1212-488c-8f06-2629a7388dc3-kube-api-access-tdcqd\") pod \"maas-api-revoke-sub-a17bdb58-rrcbv\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:27.712636 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.712614 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a17bdb58-rrcbv\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:27.712704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.712663 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4284dcb-r6dsr\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:27.712756 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.712734 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2xb9\" (UniqueName: \"kubernetes.io/projected/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-kube-api-access-z2xb9\") pod \"maas-api-revoke-sub-d4284dcb-r6dsr\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:27.789781 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.789747 4426 patch_prober.go:29] interesting pod/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:54:27.789931 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.789796 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:54:27.813894 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.813860 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a17bdb58-rrcbv\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:27.814088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.813904 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4284dcb-r6dsr\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:27.814088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.813929 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2xb9\" (UniqueName: \"kubernetes.io/projected/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-kube-api-access-z2xb9\") pod \"maas-api-revoke-sub-d4284dcb-r6dsr\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:27.814088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.813980 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a17bdb58-rrcbv\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:27.814088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.813999 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4284dcb-r6dsr\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:27.814388 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.814100 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdcqd\" (UniqueName: \"kubernetes.io/projected/311a2b24-1212-488c-8f06-2629a7388dc3-kube-api-access-tdcqd\") pod \"maas-api-revoke-sub-a17bdb58-rrcbv\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:27.814943 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.814833 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a17bdb58-rrcbv\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:27.814943 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.814886 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4284dcb-r6dsr\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:27.817137 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.817093 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a17bdb58-rrcbv\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:27.817422 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.817401 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4284dcb-r6dsr\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:27.871900 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.871827 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdcqd\" (UniqueName: \"kubernetes.io/projected/311a2b24-1212-488c-8f06-2629a7388dc3-kube-api-access-tdcqd\") pod \"maas-api-revoke-sub-a17bdb58-rrcbv\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:27.871900 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.871834 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2xb9\" (UniqueName: \"kubernetes.io/projected/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-kube-api-access-z2xb9\") pod \"maas-api-revoke-sub-d4284dcb-r6dsr\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:27.930058 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.930019 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:27.946180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.945841 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:27.986893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.986828 4426 generic.go:356] "Generic (PLEG): container finished" podID="8474f84c-56a3-4e73-8936-edd0986ab7ed" containerID="dd17f88e2adf0738671da7b175633934234af4f8c4603f71fd3626f1d3c83529" exitCode=0 Oct 31 03:54:27.987071 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:27.987003 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" event={"ID":"8474f84c-56a3-4e73-8936-edd0986ab7ed","Type":"ContainerDied","Data":"dd17f88e2adf0738671da7b175633934234af4f8c4603f71fd3626f1d3c83529"} Oct 31 03:54:28.348025 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:28.347266 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv"] Oct 31 03:54:28.398136 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:28.398090 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f6a8a5_f521_43ee_9dbd_31aaacc00f24.slice/crio-6ca6e298d4a6fa171cbb320b154908df00e851de049d0d99f5415ae0ea9692cb WatchSource:0}: Error finding container 6ca6e298d4a6fa171cbb320b154908df00e851de049d0d99f5415ae0ea9692cb: Status 404 returned error can't find the container with id 6ca6e298d4a6fa171cbb320b154908df00e851de049d0d99f5415ae0ea9692cb Oct 31 03:54:28.398453 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:28.398424 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr"] Oct 31 03:54:28.793090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:28.793058 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:28.793565 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:28.793545 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:28.794297 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:28.794274 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 03:54:28.999006 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:28.998961 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" event={"ID":"311a2b24-1212-488c-8f06-2629a7388dc3","Type":"ContainerStarted","Data":"a25eaf076a3314c8651ee99cb23787ac54584b78bb137ee45e90a833d24e5a35"} Oct 31 03:54:28.999440 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:28.999022 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" event={"ID":"311a2b24-1212-488c-8f06-2629a7388dc3","Type":"ContainerStarted","Data":"1d6de68aa484af71a91916b4b0dba42d8f3686c73115ed6e3c930d23719a153a"} Oct 31 03:54:29.000829 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.000803 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" event={"ID":"91f6a8a5-f521-43ee-9dbd-31aaacc00f24","Type":"ContainerStarted","Data":"85633e16c63d1f625ca1c2b8dda356b9f3a4bfeca87735f02c3d0838e82f88d8"} Oct 31 03:54:29.000941 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.000847 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" event={"ID":"91f6a8a5-f521-43ee-9dbd-31aaacc00f24","Type":"ContainerStarted","Data":"6ca6e298d4a6fa171cbb320b154908df00e851de049d0d99f5415ae0ea9692cb"} Oct 31 03:54:29.154261 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.152627 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" podStartSLOduration=2.152605496 podStartE2EDuration="2.152605496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:29.133521568 +0000 UTC m=+1566.134146798" watchObservedRunningTime="2026-10-31 03:54:29.152605496 +0000 UTC m=+1566.153230726" Oct 31 03:54:29.223476 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.223444 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:29.244971 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.244916 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" podStartSLOduration=2.2449018609999998 podStartE2EDuration="2.244901861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:29.241607259 +0000 UTC m=+1566.242232505" watchObservedRunningTime="2026-10-31 03:54:29.244901861 +0000 UTC m=+1566.245527089" Oct 31 03:54:29.350752 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.350710 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8474f84c-56a3-4e73-8936-edd0986ab7ed-kube-api-access-9ss8p\" (UniqueName: \"kubernetes.io/projected/8474f84c-56a3-4e73-8936-edd0986ab7ed-kube-api-access-9ss8p\") pod \"8474f84c-56a3-4e73-8936-edd0986ab7ed\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " Oct 31 03:54:29.350927 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.350876 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-xks-ca\") pod \"8474f84c-56a3-4e73-8936-edd0986ab7ed\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " Oct 31 03:54:29.351019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.350948 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-openshift-ca\") pod \"8474f84c-56a3-4e73-8936-edd0986ab7ed\" (UID: \"8474f84c-56a3-4e73-8936-edd0986ab7ed\") " Oct 31 03:54:29.351420 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.351357 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-openshift-ca" pod "8474f84c-56a3-4e73-8936-edd0986ab7ed" (UID: "8474f84c-56a3-4e73-8936-edd0986ab7ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:29.351642 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.351587 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:29.354199 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.354166 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-xks-ca" pod "8474f84c-56a3-4e73-8936-edd0986ab7ed" (UID: "8474f84c-56a3-4e73-8936-edd0986ab7ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:29.354315 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.354258 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8474f84c-56a3-4e73-8936-edd0986ab7ed-kube-api-access-9ss8p" pod "8474f84c-56a3-4e73-8936-edd0986ab7ed" (UID: "8474f84c-56a3-4e73-8936-edd0986ab7ed"). InnerVolumeSpecName "kube-api-access-9ss8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:29.453201 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.453084 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ss8p\" (UniqueName: \"kubernetes.io/projected/8474f84c-56a3-4e73-8936-edd0986ab7ed-kube-api-access-9ss8p\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:29.453201 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:29.453145 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8474f84c-56a3-4e73-8936-edd0986ab7ed-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:30.011030 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:30.010989 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" event={"ID":"8474f84c-56a3-4e73-8936-edd0986ab7ed","Type":"ContainerDied","Data":"abd8726d9633b71a041006f7964b8233eee517b8e0dc150df75725869a6ac9ea"} Oct 31 03:54:30.011501 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:30.011077 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abd8726d9633b71a041006f7964b8233eee517b8e0dc150df75725869a6ac9ea" Oct 31 03:54:30.011501 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:30.011005 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp" Oct 31 03:54:30.012994 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:30.012968 4426 generic.go:356] "Generic (PLEG): container finished" podID="91f6a8a5-f521-43ee-9dbd-31aaacc00f24" containerID="85633e16c63d1f625ca1c2b8dda356b9f3a4bfeca87735f02c3d0838e82f88d8" exitCode=0 Oct 31 03:54:30.013143 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:30.013041 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" event={"ID":"91f6a8a5-f521-43ee-9dbd-31aaacc00f24","Type":"ContainerDied","Data":"85633e16c63d1f625ca1c2b8dda356b9f3a4bfeca87735f02c3d0838e82f88d8"} Oct 31 03:54:30.014979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:30.014956 4426 generic.go:356] "Generic (PLEG): container finished" podID="311a2b24-1212-488c-8f06-2629a7388dc3" containerID="a25eaf076a3314c8651ee99cb23787ac54584b78bb137ee45e90a833d24e5a35" exitCode=0 Oct 31 03:54:30.015112 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:30.014981 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" event={"ID":"311a2b24-1212-488c-8f06-2629a7388dc3","Type":"ContainerDied","Data":"a25eaf076a3314c8651ee99cb23787ac54584b78bb137ee45e90a833d24e5a35"} Oct 31 03:54:31.240161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.240096 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:31.243672 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.243648 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:31.377877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.377796 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-xks-ca\") pod \"311a2b24-1212-488c-8f06-2629a7388dc3\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " Oct 31 03:54:31.377877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.377847 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-openshift-ca\") pod \"311a2b24-1212-488c-8f06-2629a7388dc3\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " Oct 31 03:54:31.378098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.377923 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-xks-ca\") pod \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " Oct 31 03:54:31.378098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.377976 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-kube-api-access-z2xb9\" (UniqueName: \"kubernetes.io/projected/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-kube-api-access-z2xb9\") pod \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " Oct 31 03:54:31.378098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.378016 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311a2b24-1212-488c-8f06-2629a7388dc3-kube-api-access-tdcqd\" (UniqueName: \"kubernetes.io/projected/311a2b24-1212-488c-8f06-2629a7388dc3-kube-api-access-tdcqd\") pod \"311a2b24-1212-488c-8f06-2629a7388dc3\" (UID: \"311a2b24-1212-488c-8f06-2629a7388dc3\") " Oct 31 03:54:31.378098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.378077 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-openshift-ca\") pod \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\" (UID: \"91f6a8a5-f521-43ee-9dbd-31aaacc00f24\") " Oct 31 03:54:31.378358 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.378301 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-openshift-ca" pod "311a2b24-1212-488c-8f06-2629a7388dc3" (UID: "311a2b24-1212-488c-8f06-2629a7388dc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:31.378682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.378642 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-openshift-ca" pod "91f6a8a5-f521-43ee-9dbd-31aaacc00f24" (UID: "91f6a8a5-f521-43ee-9dbd-31aaacc00f24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:31.378858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.378837 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:31.378946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.378862 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:31.380364 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.380340 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-kube-api-access-z2xb9" pod "91f6a8a5-f521-43ee-9dbd-31aaacc00f24" (UID: "91f6a8a5-f521-43ee-9dbd-31aaacc00f24"). InnerVolumeSpecName "kube-api-access-z2xb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:31.380528 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.380504 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-xks-ca" pod "311a2b24-1212-488c-8f06-2629a7388dc3" (UID: "311a2b24-1212-488c-8f06-2629a7388dc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:31.380604 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.380590 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-xks-ca" pod "91f6a8a5-f521-43ee-9dbd-31aaacc00f24" (UID: "91f6a8a5-f521-43ee-9dbd-31aaacc00f24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:31.380785 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.380771 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311a2b24-1212-488c-8f06-2629a7388dc3-kube-api-access-tdcqd" pod "311a2b24-1212-488c-8f06-2629a7388dc3" (UID: "311a2b24-1212-488c-8f06-2629a7388dc3"). InnerVolumeSpecName "kube-api-access-tdcqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:31.480725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.480681 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdcqd\" (UniqueName: \"kubernetes.io/projected/311a2b24-1212-488c-8f06-2629a7388dc3-kube-api-access-tdcqd\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:31.480725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.480723 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311a2b24-1212-488c-8f06-2629a7388dc3-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:31.480725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.480733 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:31.481019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:31.480743 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2xb9\" (UniqueName: \"kubernetes.io/projected/91f6a8a5-f521-43ee-9dbd-31aaacc00f24-kube-api-access-z2xb9\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:32.048015 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:32.047977 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" Oct 31 03:54:32.048303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:32.047986 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv" event={"ID":"311a2b24-1212-488c-8f06-2629a7388dc3","Type":"ContainerDied","Data":"1d6de68aa484af71a91916b4b0dba42d8f3686c73115ed6e3c930d23719a153a"} Oct 31 03:54:32.048402 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:32.048386 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d6de68aa484af71a91916b4b0dba42d8f3686c73115ed6e3c930d23719a153a" Oct 31 03:54:32.050372 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:32.050348 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" Oct 31 03:54:32.050485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:32.050346 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr" event={"ID":"91f6a8a5-f521-43ee-9dbd-31aaacc00f24","Type":"ContainerDied","Data":"6ca6e298d4a6fa171cbb320b154908df00e851de049d0d99f5415ae0ea9692cb"} Oct 31 03:54:32.050485 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:32.050466 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ca6e298d4a6fa171cbb320b154908df00e851de049d0d99f5415ae0ea9692cb" Oct 31 03:54:32.847852 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:54:32.847817 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:54:33.460076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.460040 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd"] Oct 31 03:54:33.460733 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.460718 4426 cpu_manager.go:414] "containerMap: removing container" podUID="91f6a8a5-f521-43ee-9dbd-31aaacc00f24" containerName="revoke-keys" Oct 31 03:54:33.460784 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.460735 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f6a8a5-f521-43ee-9dbd-31aaacc00f24" containerName="revoke-keys" Oct 31 03:54:33.460784 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.460756 4426 cpu_manager.go:414] "containerMap: removing container" podUID="8474f84c-56a3-4e73-8936-edd0986ab7ed" containerName="revoke-keys" Oct 31 03:54:33.460784 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.460761 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8474f84c-56a3-4e73-8936-edd0986ab7ed" containerName="revoke-keys" Oct 31 03:54:33.460784 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.460777 4426 cpu_manager.go:414] "containerMap: removing container" podUID="311a2b24-1212-488c-8f06-2629a7388dc3" containerName="revoke-keys" Oct 31 03:54:33.460903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.460784 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311a2b24-1212-488c-8f06-2629a7388dc3" containerName="revoke-keys" Oct 31 03:54:33.460903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.460883 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="8474f84c-56a3-4e73-8936-edd0986ab7ed" containerName="revoke-keys" Oct 31 03:54:33.460903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.460891 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="311a2b24-1212-488c-8f06-2629a7388dc3" containerName="revoke-keys" Oct 31 03:54:33.460996 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.460904 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f6a8a5-f521-43ee-9dbd-31aaacc00f24" containerName="revoke-keys" Oct 31 03:54:33.465001 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.464976 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:33.524995 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.524952 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd"] Oct 31 03:54:33.559214 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.559183 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:54:33.559456 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.559183 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:54:33.559588 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.559198 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:54:33.559841 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.559823 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:54:33.617894 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.617855 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221d193b-wkckd\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:33.618101 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.617914 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221d193b-wkckd\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:33.618101 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.618061 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p78l\" (UniqueName: \"kubernetes.io/projected/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-kube-api-access-2p78l\") pod \"maas-api-revoke-sub-221d193b-wkckd\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:33.718960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.718871 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221d193b-wkckd\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:33.718960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.718923 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221d193b-wkckd\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:33.719218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.718991 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p78l\" (UniqueName: \"kubernetes.io/projected/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-kube-api-access-2p78l\") pod \"maas-api-revoke-sub-221d193b-wkckd\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:33.719624 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.719602 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221d193b-wkckd\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:33.721497 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.721467 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221d193b-wkckd\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:33.776603 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.776571 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p78l\" (UniqueName: \"kubernetes.io/projected/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-kube-api-access-2p78l\") pod \"maas-api-revoke-sub-221d193b-wkckd\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:33.779971 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:33.779945 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:34.166772 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:34.166745 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd"] Oct 31 03:54:34.169261 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:34.169230 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6e2975_4231_4bb5_8ea1_ac39d5e8ab79.slice/crio-96a98d7343a7fe69e20e0d3b1041933117003aa1dd4410f74a71bc6641d6726b WatchSource:0}: Error finding container 96a98d7343a7fe69e20e0d3b1041933117003aa1dd4410f74a71bc6641d6726b: Status 404 returned error can't find the container with id 96a98d7343a7fe69e20e0d3b1041933117003aa1dd4410f74a71bc6641d6726b Oct 31 03:54:35.093329 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:35.093294 4426 generic.go:356] "Generic (PLEG): container finished" podID="7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79" containerID="22cceb905646fa1f049e072f5e71e78bbf59956b34019f93b0b9fa22bfa30f94" exitCode=0 Oct 31 03:54:35.093517 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:35.093382 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" event={"ID":"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79","Type":"ContainerDied","Data":"22cceb905646fa1f049e072f5e71e78bbf59956b34019f93b0b9fa22bfa30f94"} Oct 31 03:54:35.093517 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:35.093486 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" event={"ID":"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79","Type":"ContainerStarted","Data":"96a98d7343a7fe69e20e0d3b1041933117003aa1dd4410f74a71bc6641d6726b"} Oct 31 03:54:36.274918 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.274887 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:36.459105 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.459068 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-openshift-ca\") pod \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " Oct 31 03:54:36.459327 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.459155 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-xks-ca\") pod \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " Oct 31 03:54:36.459327 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.459239 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-kube-api-access-2p78l\" (UniqueName: \"kubernetes.io/projected/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-kube-api-access-2p78l\") pod \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\" (UID: \"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79\") " Oct 31 03:54:36.459585 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.459526 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-openshift-ca" pod "7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79" (UID: "7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:36.459937 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.459907 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.461641 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.461612 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-kube-api-access-2p78l" pod "7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79" (UID: "7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79"). InnerVolumeSpecName "kube-api-access-2p78l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:36.461719 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.461669 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-xks-ca" pod "7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79" (UID: "7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:36.561647 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.561613 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.561647 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.561645 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p78l\" (UniqueName: \"kubernetes.io/projected/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79-kube-api-access-2p78l\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.769331 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.769253 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p"] Oct 31 03:54:36.769941 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.769924 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79" containerName="revoke-keys" Oct 31 03:54:36.769996 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.769943 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79" containerName="revoke-keys" Oct 31 03:54:36.770048 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.770037 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79" containerName="revoke-keys" Oct 31 03:54:36.774181 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.774157 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:36.826104 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.826073 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p"] Oct 31 03:54:36.868092 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.868054 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea271435-9qk7p\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:36.868340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.868274 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea271435-9qk7p\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:36.868340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.868329 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blgmh\" (UniqueName: \"kubernetes.io/projected/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-kube-api-access-blgmh\") pod \"maas-api-revoke-sub-ea271435-9qk7p\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:36.969231 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.969189 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea271435-9qk7p\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:36.969391 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.969294 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea271435-9qk7p\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:36.969391 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.969330 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgmh\" (UniqueName: \"kubernetes.io/projected/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-kube-api-access-blgmh\") pod \"maas-api-revoke-sub-ea271435-9qk7p\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:36.969971 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.969948 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea271435-9qk7p\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:36.971739 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:36.971715 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea271435-9qk7p\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:37.029366 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:37.029291 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgmh\" (UniqueName: \"kubernetes.io/projected/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-kube-api-access-blgmh\") pod \"maas-api-revoke-sub-ea271435-9qk7p\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:37.093285 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:37.093246 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:37.115621 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:37.115594 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" Oct 31 03:54:37.115752 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:37.115592 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd" event={"ID":"7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79","Type":"ContainerDied","Data":"96a98d7343a7fe69e20e0d3b1041933117003aa1dd4410f74a71bc6641d6726b"} Oct 31 03:54:37.115752 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:37.115715 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96a98d7343a7fe69e20e0d3b1041933117003aa1dd4410f74a71bc6641d6726b" Oct 31 03:54:37.528192 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:37.528155 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p"] Oct 31 03:54:38.128106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:38.128008 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" event={"ID":"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0","Type":"ContainerStarted","Data":"fd163446db74a6e00ef7fda1651133a377bd90dc90e026ffe1930146e7d6e8b5"} Oct 31 03:54:38.128106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:38.128066 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" event={"ID":"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0","Type":"ContainerStarted","Data":"b644b44c5a0d31be9a39bf04e2264898bc9d488da2dd96bcde0775bd6882a7a9"} Oct 31 03:54:38.239777 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:38.239709 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" podStartSLOduration=2.239689197 podStartE2EDuration="2.239689197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:38.236355926 +0000 UTC m=+1575.236981154" watchObservedRunningTime="2026-10-31 03:54:38.239689197 +0000 UTC m=+1575.240314426" Oct 31 03:54:39.040944 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.040904 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8"] Oct 31 03:54:39.045931 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.045906 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.107535 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.107493 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-token\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.107715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.107590 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-certs\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.107715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.107623 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-envoy\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.107801 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.107763 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-data\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.107801 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.107794 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-credential-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.107892 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.107830 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68dg8\" (UniqueName: \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-kube-api-access-68dg8\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.107892 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.107879 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef90cb4-6f28-437e-a875-80d202b3f16c-istiod-ca-cert\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.107972 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.107896 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-podinfo\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.107972 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.107945 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.144573 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.144539 4426 generic.go:356] "Generic (PLEG): container finished" podID="6292b4c3-0129-43f1-b1f0-b78b4fd99ae0" containerID="fd163446db74a6e00ef7fda1651133a377bd90dc90e026ffe1930146e7d6e8b5" exitCode=0 Oct 31 03:54:39.144768 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.144679 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" event={"ID":"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0","Type":"ContainerDied","Data":"fd163446db74a6e00ef7fda1651133a377bd90dc90e026ffe1930146e7d6e8b5"} Oct 31 03:54:39.154667 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.154634 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8"] Oct 31 03:54:39.209055 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209022 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-data\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.209263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209063 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-credential-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.209263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209104 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68dg8\" (UniqueName: \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-kube-api-access-68dg8\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.209263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209147 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef90cb4-6f28-437e-a875-80d202b3f16c-istiod-ca-cert\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.209263 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209172 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-podinfo\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.209450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209337 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.209744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209510 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-token\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.209744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209548 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-certs\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.209744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209552 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-data\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.209744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209576 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-envoy\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.209744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209589 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-credential-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.210116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209818 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.210116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.209976 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-certs\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.210116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.210033 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef90cb4-6f28-437e-a875-80d202b3f16c-istiod-ca-cert\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.211721 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.211694 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-envoy\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.211807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.211722 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-podinfo\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.283867 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.283827 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68dg8\" (UniqueName: \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-kube-api-access-68dg8\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.284490 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.284469 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-token\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.361190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.361066 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:39.807607 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.807578 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8"] Oct 31 03:54:39.810201 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:39.810000 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef90cb4_6f28_437e_a875_80d202b3f16c.slice/crio-da2c45697996e68eb436673e78ce23a13696506faeb64052a311536a22872fd5 WatchSource:0}: Error finding container da2c45697996e68eb436673e78ce23a13696506faeb64052a311536a22872fd5: Status 404 returned error can't find the container with id da2c45697996e68eb436673e78ce23a13696506faeb64052a311536a22872fd5 Oct 31 03:54:39.813296 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.813261 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:39.813394 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.813330 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:39.813394 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.813359 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:39.826186 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.826158 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb"] Oct 31 03:54:39.832176 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.832153 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:39.860901 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.860848 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q"] Oct 31 03:54:39.865763 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.865741 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:39.889262 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.889228 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb"] Oct 31 03:54:39.934932 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.934896 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q"] Oct 31 03:54:39.935098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935033 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:39.935098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935073 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:39.935098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935093 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:39.935275 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935111 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43841623-83a4-4257-a478-a1665c421791-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:39.935275 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935155 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:39.935275 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935218 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:39.935275 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935244 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:39.935275 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935266 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43841623-83a4-4257-a478-a1665c421791-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:39.935529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935307 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:39.935529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935345 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:39.935529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935385 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:39.935529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935407 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:39.935529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935480 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:39.935782 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935598 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzbnq\" (UniqueName: \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-kube-api-access-fzbnq\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:39.935883 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935860 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:39.935999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935915 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:39.935999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935945 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlwxw\" (UniqueName: \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-kube-api-access-jlwxw\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:39.935999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:39.935972 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.007443 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.007399 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft"] Oct 31 03:54:40.012091 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.012064 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.037367 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037332 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.037367 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037370 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43841623-83a4-4257-a478-a1665c421791-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.037602 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037397 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.037602 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037435 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.037602 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037477 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.037602 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037499 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.037825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037561 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.037825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037696 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzbnq\" (UniqueName: \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-kube-api-access-fzbnq\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.037825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037764 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.037825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037800 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.037825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037819 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwxw\" (UniqueName: \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-kube-api-access-jlwxw\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.038090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037836 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.038090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037946 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.038090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037978 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.038090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037997 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.038090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.037997 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.038090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.038015 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43841623-83a4-4257-a478-a1665c421791-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.038090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.038027 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.038090 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.038053 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.038515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.038191 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.038515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.038258 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.038515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.038368 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.038515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.038450 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.039012 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.038556 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.039012 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.038885 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.039190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.039093 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.039889 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.039865 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43841623-83a4-4257-a478-a1665c421791-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.040031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.040005 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.040682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.040658 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43841623-83a4-4257-a478-a1665c421791-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.040682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.040675 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.042180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.042156 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.042551 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.042366 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.072088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.072017 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft"] Oct 31 03:54:40.096542 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.096514 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwxw\" (UniqueName: \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-kube-api-access-jlwxw\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.096814 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.096789 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.096861 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.096852 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzbnq\" (UniqueName: \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-kube-api-access-fzbnq\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.097434 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.097408 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.139748 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.139709 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-data\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.139934 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.139787 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-certs\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.140015 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.139984 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-podinfo\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.140113 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.140096 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-envoy\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.140266 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.140251 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istiod-ca-cert\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.140326 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.140288 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.140326 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.140320 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-credential-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.140465 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.140441 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-token\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.140581 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.140556 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sdx4\" (UniqueName: \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-kube-api-access-2sdx4\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.156374 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.156341 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" event={"ID":"5ef90cb4-6f28-437e-a875-80d202b3f16c","Type":"ContainerStarted","Data":"9a198853c7f27a32c63e044fdff69d060af08f0465ea99c48a1fd48a2a45c5f1"} Oct 31 03:54:40.156547 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.156402 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" event={"ID":"5ef90cb4-6f28-437e-a875-80d202b3f16c","Type":"ContainerStarted","Data":"da2c45697996e68eb436673e78ce23a13696506faeb64052a311536a22872fd5"} Oct 31 03:54:40.188625 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.188584 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:40.211162 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.211112 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:40.250252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.249543 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-token\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.250252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.249668 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sdx4\" (UniqueName: \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-kube-api-access-2sdx4\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.250252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.249742 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-data\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.250252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.249804 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-certs\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.250252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.249966 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-podinfo\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.250252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.250054 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-envoy\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.250715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.250312 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istiod-ca-cert\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.250715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.250353 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.250715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.250389 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-credential-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.251725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.250936 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-credential-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.270613 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.270570 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-data\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.270925 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.270894 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-certs\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.273824 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.273781 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istiod-ca-cert\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.277031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.276546 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-podinfo\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.291289 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.291207 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-envoy\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.293095 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.293059 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-socket\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.312365 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.312179 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podStartSLOduration=2.312159958 podStartE2EDuration="2.312159958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:40.289536359 +0000 UTC m=+1577.290161588" watchObservedRunningTime="2026-10-31 03:54:40.312159958 +0000 UTC m=+1577.312785191" Oct 31 03:54:40.319502 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.319468 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sdx4\" (UniqueName: \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-kube-api-access-2sdx4\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.344727 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.344651 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:40.348752 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.348687 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-token\") pod \"e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.362021 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.361981 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:40.365222 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.365190 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:54:40.365440 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.365393 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:54:40.453595 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.453551 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-xks-ca\") pod \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " Oct 31 03:54:40.453828 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.453804 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-openshift-ca\") pod \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " Oct 31 03:54:40.462916 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.453906 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-kube-api-access-blgmh\" (UniqueName: \"kubernetes.io/projected/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-kube-api-access-blgmh\") pod \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\" (UID: \"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0\") " Oct 31 03:54:40.462916 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.457447 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-kube-api-access-blgmh" pod "6292b4c3-0129-43f1-b1f0-b78b4fd99ae0" (UID: "6292b4c3-0129-43f1-b1f0-b78b4fd99ae0"). InnerVolumeSpecName "kube-api-access-blgmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:40.462916 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.462049 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-xks-ca" pod "6292b4c3-0129-43f1-b1f0-b78b4fd99ae0" (UID: "6292b4c3-0129-43f1-b1f0-b78b4fd99ae0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:40.462916 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.462455 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-openshift-ca" pod "6292b4c3-0129-43f1-b1f0-b78b4fd99ae0" (UID: "6292b4c3-0129-43f1-b1f0-b78b4fd99ae0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:40.560337 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.560295 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:40.560337 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.560339 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:40.560552 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.560355 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blgmh\" (UniqueName: \"kubernetes.io/projected/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0-kube-api-access-blgmh\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:40.643152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.640410 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:40.650847 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.650821 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb"] Oct 31 03:54:40.666958 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:40.666908 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43b072d_0ec4_4372_9b18_3b7c94f6902f.slice/crio-e9dae8123836a502fca22519b2c1ef44fd38e319e31ee295702a0340dc4d64dd WatchSource:0}: Error finding container e9dae8123836a502fca22519b2c1ef44fd38e319e31ee295702a0340dc4d64dd: Status 404 returned error can't find the container with id e9dae8123836a502fca22519b2c1ef44fd38e319e31ee295702a0340dc4d64dd Oct 31 03:54:40.669530 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.669489 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:40.669710 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.669686 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:40.669800 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.669744 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:40.753089 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.753057 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q"] Oct 31 03:54:40.800797 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:40.800761 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43841623_83a4_4257_a478_a1665c421791.slice/crio-9debf1a82ca140d74abfaf4bc0c0f8e24f65bb7b591ff8ac429e78f953431bca WatchSource:0}: Error finding container 9debf1a82ca140d74abfaf4bc0c0f8e24f65bb7b591ff8ac429e78f953431bca: Status 404 returned error can't find the container with id 9debf1a82ca140d74abfaf4bc0c0f8e24f65bb7b591ff8ac429e78f953431bca Oct 31 03:54:40.804484 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.804436 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:40.804604 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.804529 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:40.804604 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:40.804588 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:41.104168 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:41.103626 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3efa963_bb10_43b0_a320_8e7d9a38d0df.slice/crio-62c5a040cfd8a7ebcabf935e791636d067d9bb8be2f8d332e4be30b6026be861 WatchSource:0}: Error finding container 62c5a040cfd8a7ebcabf935e791636d067d9bb8be2f8d332e4be30b6026be861: Status 404 returned error can't find the container with id 62c5a040cfd8a7ebcabf935e791636d067d9bb8be2f8d332e4be30b6026be861 Oct 31 03:54:41.104168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.104068 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft"] Oct 31 03:54:41.107918 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.107878 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:41.108015 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.107962 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:41.108015 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.108005 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:41.188733 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.187466 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" event={"ID":"6292b4c3-0129-43f1-b1f0-b78b4fd99ae0","Type":"ContainerDied","Data":"b644b44c5a0d31be9a39bf04e2264898bc9d488da2dd96bcde0775bd6882a7a9"} Oct 31 03:54:41.188733 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.187532 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b644b44c5a0d31be9a39bf04e2264898bc9d488da2dd96bcde0775bd6882a7a9" Oct 31 03:54:41.188733 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.187603 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p" Oct 31 03:54:41.197680 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.196311 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" event={"ID":"43841623-83a4-4257-a478-a1665c421791","Type":"ContainerStarted","Data":"a2197c707bcf04689fb4299b64f4e518604d961858677c80025d167236773126"} Oct 31 03:54:41.197680 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.196376 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" event={"ID":"43841623-83a4-4257-a478-a1665c421791","Type":"ContainerStarted","Data":"9debf1a82ca140d74abfaf4bc0c0f8e24f65bb7b591ff8ac429e78f953431bca"} Oct 31 03:54:41.199447 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.199308 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" event={"ID":"f3efa963-bb10-43b0-a320-8e7d9a38d0df","Type":"ContainerStarted","Data":"62c5a040cfd8a7ebcabf935e791636d067d9bb8be2f8d332e4be30b6026be861"} Oct 31 03:54:41.203371 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.203309 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" event={"ID":"e43b072d-0ec4-4372-9b18-3b7c94f6902f","Type":"ContainerStarted","Data":"ae6253eca5fba12a780917418d266d4d7ce250cb29e92a1b1cab9d384b31f789"} Oct 31 03:54:41.203371 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.203359 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" event={"ID":"e43b072d-0ec4-4372-9b18-3b7c94f6902f","Type":"ContainerStarted","Data":"e9dae8123836a502fca22519b2c1ef44fd38e319e31ee295702a0340dc4d64dd"} Oct 31 03:54:41.226183 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.221731 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6965-15b1fc"] Oct 31 03:54:41.226183 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.221945 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6965-15b1fc" podUID="0523c1f2-0ee7-4452-b56e-92c4e792956e" containerName="test-curl-6965-15b1fc" containerID="cri-o://30de8953bce89a966991c6379c93028f8aaf45dd3b80ce2b6c2ace078ff62f8c" gracePeriod=2 Oct 31 03:54:41.226183 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.222580 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:41.258748 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.258617 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:54:41.258748 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.258677 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 03:54:41.272073 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.265026 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6965-15b1fc"] Oct 31 03:54:41.378936 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.378898 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:54:41.379103 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.378958 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:54:41.565241 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.563108 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" podStartSLOduration=2.563086289 podStartE2EDuration="2.563086289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:41.557859798 +0000 UTC m=+1578.558485028" watchObservedRunningTime="2026-10-31 03:54:41.563086289 +0000 UTC m=+1578.563711520" Oct 31 03:54:41.565241 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:41.563726 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podStartSLOduration=2.563713854 podStartE2EDuration="2.563713854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:41.341038626 +0000 UTC m=+1578.341663855" watchObservedRunningTime="2026-10-31 03:54:41.563713854 +0000 UTC m=+1578.564339083" Oct 31 03:54:41.605576 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:54:41.605544 4426 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6965-15b1fc\" is forbidden: User \"system:node:ip-10-0-134-106.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-106.us-west-2.compute.internal' and this object" podUID="0523c1f2-0ee7-4452-b56e-92c4e792956e" pod="opendatahub/test-curl-6965-15b1fc" Oct 31 03:54:42.191363 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.190885 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:42.191363 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.191027 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:54:42.191363 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.191072 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:54:42.243171 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.243137 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:54:42.243462 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.243427 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 03:54:42.312287 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.312221 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" event={"ID":"f3efa963-bb10-43b0-a320-8e7d9a38d0df","Type":"ContainerStarted","Data":"2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617"} Oct 31 03:54:42.388151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.361677 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:54:42.388151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.361751 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:54:42.556220 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.555793 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podStartSLOduration=3.555774369 podStartE2EDuration="3.555774369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:42.554592851 +0000 UTC m=+1579.555218079" watchObservedRunningTime="2026-10-31 03:54:42.555774369 +0000 UTC m=+1579.556399600" Oct 31 03:54:42.603154 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:54:42.602392 4426 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6965-15b1fc\" is forbidden: User \"system:node:ip-10-0-134-106.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-106.us-west-2.compute.internal' and this object" podUID="0523c1f2-0ee7-4452-b56e-92c4e792956e" pod="opendatahub/test-curl-6965-15b1fc" Oct 31 03:54:42.648148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.647345 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:42.648148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.647503 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 03:54:42.648148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:42.647550 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 03:54:43.080153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.077765 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:43.222149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.214117 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:54:43.222149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.214204 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:54:43.222149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.214739 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:54:43.222149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.214774 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 03:54:43.285815 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:54:43.285766 4426 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6965-15b1fc\" is forbidden: User \"system:node:ip-10-0-134-106.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-106.us-west-2.compute.internal' and this object" podUID="0523c1f2-0ee7-4452-b56e-92c4e792956e" pod="opendatahub/test-curl-6965-15b1fc" Oct 31 03:54:43.362695 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.362659 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:54:43.363007 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.362963 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:54:43.660152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.656249 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 03:54:43.660152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.656316 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 03:54:43.711152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.710224 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 31 03:54:43.743153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.742232 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:43.781803 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.775397 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:43.974151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.961596 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:43.974151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:43.962801 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 31 03:54:44.118453 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.078012 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:44.118453 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.078385 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:44.118453 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.078526 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:54:44.319707 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.294215 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:54:44.319707 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.294272 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:54:44.319707 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.319322 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:54:44.319707 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.319381 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 03:54:44.359466 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.346070 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:44.359466 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.353359 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:44.485239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.385513 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt"] Oct 31 03:54:44.485239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.386632 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6292b4c3-0129-43f1-b1f0-b78b4fd99ae0" containerName="revoke-keys" Oct 31 03:54:44.485239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.386657 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6292b4c3-0129-43f1-b1f0-b78b4fd99ae0" containerName="revoke-keys" Oct 31 03:54:44.485239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.386705 4426 cpu_manager.go:414] "containerMap: removing container" podUID="0523c1f2-0ee7-4452-b56e-92c4e792956e" containerName="test-curl-6965-15b1fc" Oct 31 03:54:44.485239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.386714 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0523c1f2-0ee7-4452-b56e-92c4e792956e" containerName="test-curl-6965-15b1fc" Oct 31 03:54:44.485239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.386877 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="6292b4c3-0129-43f1-b1f0-b78b4fd99ae0" containerName="revoke-keys" Oct 31 03:54:44.485239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.386899 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="0523c1f2-0ee7-4452-b56e-92c4e792956e" containerName="test-curl-6965-15b1fc" Oct 31 03:54:44.485239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.452796 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:54:44.485239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.452854 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:54:44.711299 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.711259 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:44.742529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.713006 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:44.807198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.715818 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 03:54:44.807465 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.807438 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 03:54:44.822840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.749066 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:44.842452 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.757227 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:44.852550 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.763148 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:54:44.878150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.763231 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:54:44.878405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.763273 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:54:44.878527 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.763353 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:54:44.878612 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.788619 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:44.879741 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.796219 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:44.906929 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.807231 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:54:44.906929 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.902638 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:54:44.906929 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.902664 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt"] Oct 31 03:54:44.933148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.932292 4426 generic.go:356] "Generic (PLEG): container finished" podID="0523c1f2-0ee7-4452-b56e-92c4e792956e" containerID="30de8953bce89a966991c6379c93028f8aaf45dd3b80ce2b6c2ace078ff62f8c" exitCode=137 Oct 31 03:54:44.933148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.749022 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 03:54:44.933148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:44.932911 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b474ce3fc818c769e6326bbf16d41a1aeb72f021f50107ef7fd27cf56fb77e36" Oct 31 03:54:45.069196 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.069156 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Oct 31 03:54:45.116617 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.116572 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:54:45.138266 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.138230 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1538000c-913d-4117-92b4-d2549581a428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93800a09-lc9gt\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:45.138811 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.138773 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vdk5\" (UniqueName: \"kubernetes.io/projected/1538000c-913d-4117-92b4-d2549581a428-kube-api-access-8vdk5\") pod \"maas-api-revoke-sub-93800a09-lc9gt\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:45.139243 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.139225 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1538000c-913d-4117-92b4-d2549581a428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93800a09-lc9gt\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:45.141975 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.141945 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:45.164220 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.164184 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-15b1fc" Oct 31 03:54:45.190220 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.190184 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:54:45.190508 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.190475 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:54:45.219728 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.219695 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:54:45.219989 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.219962 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 03:54:45.237290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.237244 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/health\": context deadline exceeded" Oct 31 03:54:45.252430 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.252390 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0523c1f2-0ee7-4452-b56e-92c4e792956e-kube-api-access-l2kcn\" (UniqueName: \"kubernetes.io/projected/0523c1f2-0ee7-4452-b56e-92c4e792956e-kube-api-access-l2kcn\") pod \"0523c1f2-0ee7-4452-b56e-92c4e792956e\" (UID: \"0523c1f2-0ee7-4452-b56e-92c4e792956e\") " Oct 31 03:54:45.278206 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.253280 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1538000c-913d-4117-92b4-d2549581a428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93800a09-lc9gt\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:45.279250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.279211 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vdk5\" (UniqueName: \"kubernetes.io/projected/1538000c-913d-4117-92b4-d2549581a428-kube-api-access-8vdk5\") pod \"maas-api-revoke-sub-93800a09-lc9gt\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:45.279651 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.279630 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1538000c-913d-4117-92b4-d2549581a428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93800a09-lc9gt\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:45.280566 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.269094 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1538000c-913d-4117-92b4-d2549581a428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93800a09-lc9gt\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:45.367765 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.367723 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:45.428151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.383529 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:45.428151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.383593 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:45.461149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.460420 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:54:45.461149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.460491 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:54:45.668194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.668157 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0523c1f2-0ee7-4452-b56e-92c4e792956e-kube-api-access-l2kcn" pod "0523c1f2-0ee7-4452-b56e-92c4e792956e" (UID: "0523c1f2-0ee7-4452-b56e-92c4e792956e"). InnerVolumeSpecName "kube-api-access-l2kcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:45.697716 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.697680 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1538000c-913d-4117-92b4-d2549581a428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93800a09-lc9gt\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:45.727150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.713308 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 03:54:45.727150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.713374 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 03:54:45.727150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.724105 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2kcn\" (UniqueName: \"kubernetes.io/projected/0523c1f2-0ee7-4452-b56e-92c4e792956e-kube-api-access-l2kcn\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:45.755530 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.754818 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:45.778567 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.760450 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vdk5\" (UniqueName: \"kubernetes.io/projected/1538000c-913d-4117-92b4-d2549581a428-kube-api-access-8vdk5\") pod \"maas-api-revoke-sub-93800a09-lc9gt\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:45.795639 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.794500 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:45.822403 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.822365 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:45.823733 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.823680 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:45.839146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.839094 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:45.945212 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.938159 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:45.949523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.949481 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:54:45.949995 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.949966 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Oct 31 03:54:45.960161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:45.956379 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:46.042970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.042934 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0523c1f2-0ee7-4452-b56e-92c4e792956e" path="/var/lib/kubelet/pods/0523c1f2-0ee7-4452-b56e-92c4e792956e/volumes" Oct 31 03:54:46.132422 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.131379 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:46.224244 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:54:46.223207 4426 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.057s" Oct 31 03:54:46.224244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.223426 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:54:46.224244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.223498 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:54:46.224244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.223549 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:54:46.224244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.223605 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:54:46.224244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.223660 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:54:46.224244 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.223709 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:54:46.279161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.276316 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:46.325152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.313770 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:54:46.325152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.313852 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 03:54:46.346151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.331184 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:54:46.346521 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.346483 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:54:46.454181 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.442288 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:46.454181 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.451368 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:46.573258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.565836 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:54:46.573258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.565915 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:54:46.799064 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.799029 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-15b1fc" Oct 31 03:54:46.970276 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.970232 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 31 03:54:46.972024 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.971998 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:54:46.984834 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:46.975023 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 03:54:47.163461 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.005003 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:47.163952 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.005178 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:47.179585 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.005215 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 31 03:54:47.180098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.006093 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:54:47.180384 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.180364 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:54:47.211666 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.006298 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:47.246153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.006626 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:47.366307 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.365197 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Oct 31 03:54:47.370184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.006572 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 03:54:47.370184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.368246 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 03:54:47.370184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.220404 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:54:47.370184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.368299 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:54:47.370184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.367917 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:54:47.370184 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.368336 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 03:54:47.401601 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.396593 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:54:47.415407 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.415311 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:54:47.442148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.433893 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:47.465511 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.448429 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:47.468109 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.468066 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:47.469963 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.469918 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:47.912904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.904761 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 03:54:47.912904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:47.904819 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 03:54:48.004043 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.004010 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:54:48.009435 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.009395 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:48.029404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.029349 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:48.154283 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.154237 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/health\": context deadline exceeded" Oct 31 03:54:48.207226 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.206991 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:54:48.207226 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.207054 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:54:48.210366 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.210303 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:48.211227 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.210692 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:48.246223 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.212246 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:54:48.247501 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.213150 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:54:48.247501 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.247228 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 03:54:48.258432 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.256272 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:48.310160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.309857 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Oct 31 03:54:48.394355 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.372222 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:48.394355 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.389251 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Oct 31 03:54:48.434591 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.434553 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:54:48.434799 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.434631 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:48.490578 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.464262 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:54:48.490578 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.464325 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:54:48.547153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.535068 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:48.579275 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.568268 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 31 03:54:48.698305 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.678724 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 03:54:48.698305 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:48.678800 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 03:54:49.014278 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.014231 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:54:49.040214 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.014935 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:54:49.047158 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.040529 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:49.096662 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.095656 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:54:49.197150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.189566 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:54:49.197150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.189630 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:54:49.252151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.219876 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:54:49.252151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.219935 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 03:54:49.252151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.220001 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 03:54:49.252151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.220311 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:49.279708 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.279662 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:49.311336 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.311217 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:54:49.417152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.391662 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:54:49.417493 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.417451 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:54:49.516322 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.516278 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Oct 31 03:54:49.591664 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.533760 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:49.592118 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.552424 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:49.599520 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.587976 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 31 03:54:49.769148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.759994 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 03:54:49.824157 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:49.802362 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 03:54:50.213150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.199236 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:54:50.302023 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.235144 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:50.302023 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.235357 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:50.302023 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.239526 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 31 03:54:50.302023 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.281619 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:50.309221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.302550 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:54:50.309221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.302594 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 03:54:50.316310 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.316278 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:54:50.316542 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.316516 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:54:50.403239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.397833 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:54:50.481376 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.471304 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:50.481376 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.472990 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:54:50.544565 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.544532 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:54:50.545650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.545173 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:54:50.545650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.545239 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:54:50.545650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.545291 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:54:50.579908 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.579861 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:54:50.623169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.587158 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" event={"ID":"1538000c-913d-4117-92b4-d2549581a428","Type":"ContainerStarted","Data":"f8419498ea8ec27ac0acdc7729bb04b0317a078e6a4d2aab9427fc2ab1c12d42"} Oct 31 03:54:50.623169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.623161 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:54:50.624170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.615521 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:50.628151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.627849 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:54:50.629162 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.628896 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:54:50.646141 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.646087 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 03:54:50.646703 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.646658 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 03:54:50.804000 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.803917 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:54:50.812256 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.812230 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:54:50.848840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.848760 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:50.850559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.850541 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:50.869921 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.869849 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 03:54:50.884264 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.882542 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r"] Oct 31 03:54:50.920843 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.920801 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:50.926216 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.926183 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r"] Oct 31 03:54:50.939723 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.939688 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f1c6bce-rgs5r\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:50.945591 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.945540 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f1c6bce-rgs5r\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:50.945923 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.945900 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qngvb\" (UniqueName: \"kubernetes.io/projected/1c94b670-dab9-4ddc-af46-c6565dd0d16d-kube-api-access-qngvb\") pod \"maas-api-revoke-sub-6f1c6bce-rgs5r\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:50.987822 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:50.987790 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt"] Oct 31 03:54:51.048735 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.047688 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f1c6bce-rgs5r\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:51.048735 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.047845 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f1c6bce-rgs5r\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:51.048735 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.047925 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngvb\" (UniqueName: \"kubernetes.io/projected/1c94b670-dab9-4ddc-af46-c6565dd0d16d-kube-api-access-qngvb\") pod \"maas-api-revoke-sub-6f1c6bce-rgs5r\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:51.049615 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.049588 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f1c6bce-rgs5r\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:51.062437 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.061398 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f1c6bce-rgs5r\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:51.115754 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.114043 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qngvb\" (UniqueName: \"kubernetes.io/projected/1c94b670-dab9-4ddc-af46-c6565dd0d16d-kube-api-access-qngvb\") pod \"maas-api-revoke-sub-6f1c6bce-rgs5r\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:51.211686 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.211570 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:54:51.211686 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.211628 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 03:54:51.284400 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.284370 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:51.671438 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.642183 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" event={"ID":"1538000c-913d-4117-92b4-d2549581a428","Type":"ContainerStarted","Data":"15e95a3adadcefe7bc627bd5a173fad3385c105cda5c74ac92651eaef45bb9d0"} Oct 31 03:54:51.671438 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.644472 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 03:54:51.671438 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:51.644534 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 03:54:52.097534 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:52.090799 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" podStartSLOduration=8.090780959 podStartE2EDuration="8.090780959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:52.090093801 +0000 UTC m=+1589.090719030" watchObservedRunningTime="2026-10-31 03:54:52.090780959 +0000 UTC m=+1589.091406188" Oct 31 03:54:52.165219 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:52.153545 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:52.165219 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:52.155069 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:52.229160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:52.226441 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 03:54:52.871228 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:52.871187 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c94b670_dab9_4ddc_af46_c6565dd0d16d.slice/crio-b38f2b8786559e8addfc6058c747e669ce5e1b63cde11d133cea3c1964117ed2 WatchSource:0}: Error finding container b38f2b8786559e8addfc6058c747e669ce5e1b63cde11d133cea3c1964117ed2: Status 404 returned error can't find the container with id b38f2b8786559e8addfc6058c747e669ce5e1b63cde11d133cea3c1964117ed2 Oct 31 03:54:52.908460 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:52.908413 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" event={"ID":"1538000c-913d-4117-92b4-d2549581a428","Type":"ContainerDied","Data":"15e95a3adadcefe7bc627bd5a173fad3385c105cda5c74ac92651eaef45bb9d0"} Oct 31 03:54:52.908733 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:52.908587 4426 generic.go:356] "Generic (PLEG): container finished" podID="1538000c-913d-4117-92b4-d2549581a428" containerID="15e95a3adadcefe7bc627bd5a173fad3385c105cda5c74ac92651eaef45bb9d0" exitCode=0 Oct 31 03:54:52.963419 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:52.962321 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r"] Oct 31 03:54:53.017043 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:53.017009 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:53.019205 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:53.019177 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:53.024816 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:53.021399 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 03:54:53.292149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:53.291269 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:54:53.292149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:53.291348 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:53.445047 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:53.445004 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:53.446655 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:53.446631 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:53.448102 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:53.448072 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 03:54:53.922943 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:53.922893 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" event={"ID":"1c94b670-dab9-4ddc-af46-c6565dd0d16d","Type":"ContainerStarted","Data":"48f39c9c49e317a70ec43d26feb25fed5294a750b5cd622c938ef2c1b8679f49"} Oct 31 03:54:53.923418 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:53.922977 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" event={"ID":"1c94b670-dab9-4ddc-af46-c6565dd0d16d","Type":"ContainerStarted","Data":"b38f2b8786559e8addfc6058c747e669ce5e1b63cde11d133cea3c1964117ed2"} Oct 31 03:54:54.122384 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.122347 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:54.150094 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.150034 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" podStartSLOduration=6.150016288 podStartE2EDuration="6.150016288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:54.036862272 +0000 UTC m=+1591.037487501" watchObservedRunningTime="2026-10-31 03:54:54.150016288 +0000 UTC m=+1591.150641565" Oct 31 03:54:54.152107 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.152079 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m"] Oct 31 03:54:54.152875 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.152858 4426 cpu_manager.go:414] "containerMap: removing container" podUID="1538000c-913d-4117-92b4-d2549581a428" containerName="revoke-keys" Oct 31 03:54:54.152920 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.152877 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1538000c-913d-4117-92b4-d2549581a428" containerName="revoke-keys" Oct 31 03:54:54.153042 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.153030 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="1538000c-913d-4117-92b4-d2549581a428" containerName="revoke-keys" Oct 31 03:54:54.157337 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.157314 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.200621 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.200591 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1538000c-913d-4117-92b4-d2549581a428-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1538000c-913d-4117-92b4-d2549581a428-maas-api-openshift-ca\") pod \"1538000c-913d-4117-92b4-d2549581a428\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " Oct 31 03:54:54.200809 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.200739 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1538000c-913d-4117-92b4-d2549581a428-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1538000c-913d-4117-92b4-d2549581a428-maas-api-xks-ca\") pod \"1538000c-913d-4117-92b4-d2549581a428\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " Oct 31 03:54:54.200809 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.200792 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1538000c-913d-4117-92b4-d2549581a428-kube-api-access-8vdk5\" (UniqueName: \"kubernetes.io/projected/1538000c-913d-4117-92b4-d2549581a428-kube-api-access-8vdk5\") pod \"1538000c-913d-4117-92b4-d2549581a428\" (UID: \"1538000c-913d-4117-92b4-d2549581a428\") " Oct 31 03:54:54.200960 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.200917 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1538000c-913d-4117-92b4-d2549581a428-maas-api-openshift-ca" pod "1538000c-913d-4117-92b4-d2549581a428" (UID: "1538000c-913d-4117-92b4-d2549581a428"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:54.201093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.201074 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.201214 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.201195 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-data\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.201283 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.201259 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-envoy\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.201342 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.201295 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istiod-ca-cert\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.201465 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.201442 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjq6z\" (UniqueName: \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-kube-api-access-zjq6z\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.201575 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.201556 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-token\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.201672 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.201654 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-certs\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.201729 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.201706 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-credential-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.201858 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.201841 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-podinfo\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.201999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.201978 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1538000c-913d-4117-92b4-d2549581a428-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:54.203450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.203416 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1538000c-913d-4117-92b4-d2549581a428-maas-api-xks-ca" pod "1538000c-913d-4117-92b4-d2549581a428" (UID: "1538000c-913d-4117-92b4-d2549581a428"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:54.203548 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.203463 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1538000c-913d-4117-92b4-d2549581a428-kube-api-access-8vdk5" pod "1538000c-913d-4117-92b4-d2549581a428" (UID: "1538000c-913d-4117-92b4-d2549581a428"). InnerVolumeSpecName "kube-api-access-8vdk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:54.214813 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.214773 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m"] Oct 31 03:54:54.303966 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.303921 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304272 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.303986 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-data\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304272 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304014 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-envoy\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304272 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304042 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istiod-ca-cert\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304272 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304204 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjq6z\" (UniqueName: \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-kube-api-access-zjq6z\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304263 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-token\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304347 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-certs\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304375 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-credential-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304420 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304441 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-podinfo\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304455 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-data\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304620 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1538000c-913d-4117-92b4-d2549581a428-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:54.304825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304642 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vdk5\" (UniqueName: \"kubernetes.io/projected/1538000c-913d-4117-92b4-d2549581a428-kube-api-access-8vdk5\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:54.304825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304695 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-certs\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304774 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-credential-socket\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.304992 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.304857 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istiod-ca-cert\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.306909 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.306881 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-envoy\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.307104 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.307083 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-podinfo\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.376786 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.376716 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-token\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.381404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.381362 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjq6z\" (UniqueName: \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-kube-api-access-zjq6z\") pod \"e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.474702 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.474607 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:54.937437 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.936731 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m"] Oct 31 03:54:54.940242 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:54.940192 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba30cf5_bad6_4938_a1d4_c125dc8ac20e.slice/crio-703a34d938cdf59ce9edba4f68f84c54ff7c7a147fb1262c636753e4ef5d136f WatchSource:0}: Error finding container 703a34d938cdf59ce9edba4f68f84c54ff7c7a147fb1262c636753e4ef5d136f: Status 404 returned error can't find the container with id 703a34d938cdf59ce9edba4f68f84c54ff7c7a147fb1262c636753e4ef5d136f Oct 31 03:54:54.942747 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.942717 4426 generic.go:356] "Generic (PLEG): container finished" podID="1c94b670-dab9-4ddc-af46-c6565dd0d16d" containerID="48f39c9c49e317a70ec43d26feb25fed5294a750b5cd622c938ef2c1b8679f49" exitCode=0 Oct 31 03:54:54.942964 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.942935 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" event={"ID":"1c94b670-dab9-4ddc-af46-c6565dd0d16d","Type":"ContainerDied","Data":"48f39c9c49e317a70ec43d26feb25fed5294a750b5cd622c938ef2c1b8679f49"} Oct 31 03:54:54.943562 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.943523 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:54.944405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.944356 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:54.944602 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.944575 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:54.953623 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.953588 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" Oct 31 03:54:54.954080 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.954033 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt" event={"ID":"1538000c-913d-4117-92b4-d2549581a428","Type":"ContainerDied","Data":"f8419498ea8ec27ac0acdc7729bb04b0317a078e6a4d2aab9427fc2ab1c12d42"} Oct 31 03:54:54.954233 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.954211 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8419498ea8ec27ac0acdc7729bb04b0317a078e6a4d2aab9427fc2ab1c12d42" Oct 31 03:54:54.954690 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.954651 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr"] Oct 31 03:54:54.971173 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:54.971103 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.021379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.018633 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr"] Oct 31 03:54:55.021379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.019872 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.021379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.020015 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.021379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.020086 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.021379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.020143 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.021379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.020301 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebd34aa-f849-416e-9a2a-b895527619a8-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.021379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.020400 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.021379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.020532 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.021379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.020584 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vsrg\" (UniqueName: \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-kube-api-access-9vsrg\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.021379 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.020635 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.122559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.122522 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.122834 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.122810 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vsrg\" (UniqueName: \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-kube-api-access-9vsrg\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.123017 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.123000 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.123266 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.123219 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.123528 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.123511 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.123698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.123685 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.123816 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.123801 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.124034 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.124013 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebd34aa-f849-416e-9a2a-b895527619a8-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.124238 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.124218 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.124454 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.124118 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-certs\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.124884 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.124856 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-data\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.125503 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.125479 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebd34aa-f849-416e-9a2a-b895527619a8-istiod-ca-cert\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.128628 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.127696 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-credential-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.128628 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.128008 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-socket\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.128628 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.128563 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-podinfo\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.134189 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.129895 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-envoy\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.181396 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.181358 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vsrg\" (UniqueName: \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-kube-api-access-9vsrg\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.184357 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.184326 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-token\") pod \"e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.287730 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.287640 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:55.752427 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.752398 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr"] Oct 31 03:54:55.753894 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:54:55.753869 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebd34aa_f849_416e_9a2a_b895527619a8.slice/crio-737f6895affdb037444f403ccfb88f6876878df7cd75dd69c045743e45c7382c WatchSource:0}: Error finding container 737f6895affdb037444f403ccfb88f6876878df7cd75dd69c045743e45c7382c: Status 404 returned error can't find the container with id 737f6895affdb037444f403ccfb88f6876878df7cd75dd69c045743e45c7382c Oct 31 03:54:55.756654 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.756611 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:55.756742 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.756713 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:55.756781 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:55.756754 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:54:56.011415 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.010911 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" event={"ID":"4ebd34aa-f849-416e-9a2a-b895527619a8","Type":"ContainerStarted","Data":"11faa1f8d8a95f2ddbc05de14401bb6a0bdd07fb98960fd7f9819c2ad3198d8b"} Oct 31 03:54:56.012347 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.012320 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" event={"ID":"4ebd34aa-f849-416e-9a2a-b895527619a8","Type":"ContainerStarted","Data":"737f6895affdb037444f403ccfb88f6876878df7cd75dd69c045743e45c7382c"} Oct 31 03:54:56.033497 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.033397 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" event={"ID":"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e","Type":"ContainerStarted","Data":"d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7"} Oct 31 03:54:56.033791 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.033761 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" event={"ID":"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e","Type":"ContainerStarted","Data":"703a34d938cdf59ce9edba4f68f84c54ff7c7a147fb1262c636753e4ef5d136f"} Oct 31 03:54:56.162346 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.162281 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podStartSLOduration=2.162255063 podStartE2EDuration="2.162255063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:56.145400601 +0000 UTC m=+1593.146025830" watchObservedRunningTime="2026-10-31 03:54:56.162255063 +0000 UTC m=+1593.162880292" Oct 31 03:54:56.278770 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.278656 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podStartSLOduration=2.278638027 podStartE2EDuration="2.278638027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:56.271792029 +0000 UTC m=+1593.272417257" watchObservedRunningTime="2026-10-31 03:54:56.278638027 +0000 UTC m=+1593.279263256" Oct 31 03:54:56.289437 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.289189 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:54:56.289598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.289431 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:54:56.289598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.289479 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:54:56.352683 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.352224 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:56.375440 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.375400 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-openshift-ca\") pod \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " Oct 31 03:54:56.375630 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.375614 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-xks-ca\") pod \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " Oct 31 03:54:56.375742 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.375726 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c94b670-dab9-4ddc-af46-c6565dd0d16d-kube-api-access-qngvb\" (UniqueName: \"kubernetes.io/projected/1c94b670-dab9-4ddc-af46-c6565dd0d16d-kube-api-access-qngvb\") pod \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\" (UID: \"1c94b670-dab9-4ddc-af46-c6565dd0d16d\") " Oct 31 03:54:56.375790 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.375741 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-openshift-ca" pod "1c94b670-dab9-4ddc-af46-c6565dd0d16d" (UID: "1c94b670-dab9-4ddc-af46-c6565dd0d16d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:56.376328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.376302 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:56.377870 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.377842 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c94b670-dab9-4ddc-af46-c6565dd0d16d-kube-api-access-qngvb" pod "1c94b670-dab9-4ddc-af46-c6565dd0d16d" (UID: "1c94b670-dab9-4ddc-af46-c6565dd0d16d"). InnerVolumeSpecName "kube-api-access-qngvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:56.377999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.377937 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-xks-ca" pod "1c94b670-dab9-4ddc-af46-c6565dd0d16d" (UID: "1c94b670-dab9-4ddc-af46-c6565dd0d16d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:56.479788 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.479545 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c94b670-dab9-4ddc-af46-c6565dd0d16d-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:56.479788 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.479581 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qngvb\" (UniqueName: \"kubernetes.io/projected/1c94b670-dab9-4ddc-af46-c6565dd0d16d-kube-api-access-qngvb\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:56.479788 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.479616 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:54:56.480596 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.480570 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:54:56.480734 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:56.480623 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:54:57.065148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:57.064663 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" Oct 31 03:54:57.069144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:57.068272 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r" event={"ID":"1c94b670-dab9-4ddc-af46-c6565dd0d16d","Type":"ContainerDied","Data":"b38f2b8786559e8addfc6058c747e669ce5e1b63cde11d133cea3c1964117ed2"} Oct 31 03:54:57.069144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:57.068340 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b38f2b8786559e8addfc6058c747e669ce5e1b63cde11d133cea3c1964117ed2" Oct 31 03:54:57.308149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:57.307692 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:54:57.308149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:57.307762 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:54:57.489152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:57.481522 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:54:57.489152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:57.481586 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:54:58.186772 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.186230 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:58.186772 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.186694 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:58.251286 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.251232 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:58.251999 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.251545 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:58.307151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.288410 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:54:58.307151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.288475 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:54:58.312076 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.311223 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:58.312743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.312368 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 03:54:58.339814 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.339344 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:54:58.391154 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.388275 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:58.443611 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.443510 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 31 03:54:58.445403 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.444703 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:58.479153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.477281 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:54:58.479153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.477360 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:54:58.523290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.521250 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:58.523290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.522571 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:58.897890 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:58.886586 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:59.108169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.108105 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 31 03:54:59.259210 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.192800 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:59.259210 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.193229 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:54:59.272150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.265356 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:59.274067 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.274028 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 03:54:59.309284 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.309244 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:54:59.310241 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.310201 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:54:59.339631 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.313676 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:59.364050 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.313709 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Oct 31 03:54:59.364050 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.344985 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:54:59.507161 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.505057 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 31 03:54:59.617549 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.543271 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Oct 31 03:54:59.627151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.571248 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:54:59.627643 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.571285 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 31 03:54:59.762214 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.762180 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:54:59.762478 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.762447 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:54:59.891214 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.890242 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 31 03:54:59.944153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.944092 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:54:59.966359 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.966307 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:54:59.966739 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.966716 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:54:59.966923 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.966909 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:54:59.967070 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.967058 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:54:59.967212 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.967197 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf"] Oct 31 03:54:59.968442 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.968413 4426 cpu_manager.go:414] "containerMap: removing container" podUID="1c94b670-dab9-4ddc-af46-c6565dd0d16d" containerName="revoke-keys" Oct 31 03:54:59.968618 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.968601 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c94b670-dab9-4ddc-af46-c6565dd0d16d" containerName="revoke-keys" Oct 31 03:54:59.969052 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:54:59.969034 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c94b670-dab9-4ddc-af46-c6565dd0d16d" containerName="revoke-keys" Oct 31 03:55:00.113877 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.113834 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 31 03:55:00.362354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.268468 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:55:00.370386 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.370286 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:55:00.379267 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.374290 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:55:00.403790 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.268560 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:55:00.404039 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.356383 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:00.404258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.358447 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:00.404432 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.359315 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:55:00.404567 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.404542 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:00.404680 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.359453 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Oct 31 03:55:00.404803 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.360619 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": context deadline exceeded" Oct 31 03:55:00.404924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.359358 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:00.405054 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.360699 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:55:00.405198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.405176 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:55:00.405366 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.295570 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f991c4d-a12d-41b1-af5b-d806c2441ddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf\" (UID: \"4f991c4d-a12d-41b1-af5b-d806c2441ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:55:00.405689 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.405654 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzw85\" (UniqueName: \"kubernetes.io/projected/4f991c4d-a12d-41b1-af5b-d806c2441ddf-kube-api-access-lzw85\") pod \"payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf\" (UID: \"4f991c4d-a12d-41b1-af5b-d806c2441ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:55:00.406447 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.356335 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Oct 31 03:55:00.512302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.512262 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:55:00.516268 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.516226 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf"] Oct 31 03:55:00.571300 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.571255 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:55:00.613393 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.613296 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:55:00.687151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.572581 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f991c4d-a12d-41b1-af5b-d806c2441ddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf\" (UID: \"4f991c4d-a12d-41b1-af5b-d806c2441ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:55:00.687151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.643450 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzw85\" (UniqueName: \"kubernetes.io/projected/4f991c4d-a12d-41b1-af5b-d806c2441ddf-kube-api-access-lzw85\") pod \"payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf\" (UID: \"4f991c4d-a12d-41b1-af5b-d806c2441ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:55:00.687151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.620240 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:00.687151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.645495 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Oct 31 03:55:00.687151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.645804 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:55:00.687151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.646077 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 31 03:55:00.720077 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.720037 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:55:00.799289 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.799241 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:55:00.799571 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.722255 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-34a72a1f\"" Oct 31 03:55:00.840222 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.768102 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f991c4d-a12d-41b1-af5b-d806c2441ddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf\" (UID: \"4f991c4d-a12d-41b1-af5b-d806c2441ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:55:00.840489 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.722190 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-34a72a1f-dockercfg-2f5x8\"" Oct 31 03:55:00.866207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.855966 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzw85\" (UniqueName: \"kubernetes.io/projected/4f991c4d-a12d-41b1-af5b-d806c2441ddf-kube-api-access-lzw85\") pod \"payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf\" (UID: \"4f991c4d-a12d-41b1-af5b-d806c2441ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:55:00.891234 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:00.891189 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:01.385346 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.385294 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 31 03:55:01.464151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.417438 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:55:01.464151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.417512 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:55:01.491154 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.465626 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:55:01.545163 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.524232 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:01.556164 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.553460 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:01.556164 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.553625 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Oct 31 03:55:01.636175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.636033 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:55:01.636470 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.636433 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:01.689908 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.689852 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:01.766863 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.766814 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Oct 31 03:55:02.392150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.794200 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": context deadline exceeded" Oct 31 03:55:02.393103 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.794554 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 03:55:02.414107 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.799015 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:55:02.414676 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.820313 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 31 03:55:02.417924 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.820501 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Oct 31 03:55:02.418337 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.822099 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 31 03:55:02.418673 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.870043 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/health\": context deadline exceeded" Oct 31 03:55:02.427157 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:01.970628 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:55:02.449054 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.449007 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:55:02.449340 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:55:02.191183 4426 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.023s" Oct 31 03:55:02.450096 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.154542 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh" podUID="c785f917-ad62-4bcf-9b30-dcbebdbda03b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:02.483004 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.451950 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:02.483490 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.452001 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 31 03:55:02.483814 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.480778 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:02.515153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.495781 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:55:02.515153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.495860 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:55:02.551910 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.551865 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:02.584047 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.552234 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:55:02.584426 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.584389 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:02.584598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.558550 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Oct 31 03:55:02.586774 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.558493 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:02.677370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.677269 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:55:02.677680 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.677648 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:55:02.708031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:02.707972 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:03.256159 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.249243 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:55:03.273205 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.260239 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:03.490160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.489169 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:55:03.490160 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.489237 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:55:03.519635 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.497736 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:03.519635 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.498154 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Oct 31 03:55:03.519635 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.498461 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": context deadline exceeded" Oct 31 03:55:03.519635 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.498703 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 03:55:03.670790 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.669381 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Oct 31 03:55:03.670790 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.669774 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 03:55:03.795466 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:55:03.795348 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:55:03.856153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.830841 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 31 03:55:04.135821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:03.843453 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 31 03:55:04.233679 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.090087 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:55:04.233990 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.233949 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:55:04.238397 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.192180 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/health\": context deadline exceeded" Oct 31 03:55:04.369155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.368252 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:04.369155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.368362 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:04.388217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.370334 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 31 03:55:04.388217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.371219 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 31 03:55:04.516467 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.516422 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:55:04.522236 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.522193 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:04.607394 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.529419 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:04.607653 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.531334 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:04.607776 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.531398 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:55:04.607886 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.531555 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:04.739302 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.739254 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:04.739651 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.739614 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Oct 31 03:55:04.896219 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:04.883243 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:05.003024 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:55:05.002983 4426 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.873s" Oct 31 03:55:05.021255 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:05.010460 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:55:05.021255 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:05.010517 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:55:05.021255 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:05.010604 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:55:05.021255 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:05.010627 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:55:05.449528 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:05.438994 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/health\": context deadline exceeded" Oct 31 03:55:05.449528 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:05.440375 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 31 03:55:05.483194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:05.482761 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:05.491912 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:05.491879 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:55:05.492228 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:05.492175 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:55:05.529746 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:05.527245 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:55:05.529746 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:05.527305 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:55:06.009100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.009045 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:06.009411 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.009389 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:55:06.010834 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.010785 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Oct 31 03:55:06.012080 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.012040 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 31 03:55:06.033314 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.033273 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 31 03:55:06.034588 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.034551 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 03:55:06.035891 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.035857 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Oct 31 03:55:06.036619 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.036587 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:06.037815 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.037775 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:06.065521 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.057445 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Oct 31 03:55:06.111781 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.111744 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 03:55:06.301455 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.301418 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 03:55:06.335188 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.311773 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:55:06.335188 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.311851 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:55:06.335188 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.316713 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" event={"ID":"4f991c4d-a12d-41b1-af5b-d806c2441ddf","Type":"ContainerStarted","Data":"c94a124466850e1f368d0940db912ebde31437d3cce30f024440b3d4fad8b622"} Oct 31 03:55:06.609306 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.609239 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:06.611000 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.610968 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 31 03:55:06.667270 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.667232 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:55:06.768784 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.768732 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:55:06.769339 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:06.757940 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 03:55:07.286148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.272477 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Oct 31 03:55:07.293144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.053799 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:07.293144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.121735 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:07.298144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.213066 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:07.298144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.121671 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:07.341159 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.298427 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:55:07.341159 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.298488 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:55:07.341159 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.304198 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:07.341159 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.284023 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 03:55:07.472674 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.472364 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 03:55:07.472674 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.472503 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 03:55:07.472674 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.472539 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf"] Oct 31 03:55:07.501657 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.501620 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:55:07.501973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.501942 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:55:07.505103 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.505075 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 03:55:07.505283 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.505150 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 03:55:07.505283 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.505184 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 03:55:07.507003 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.506981 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 03:55:07.515140 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.507628 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 03:55:07.583627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.583535 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 03:55:07.588544 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.588511 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 03:55:07.596170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.593727 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 03:55:07.596170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.593767 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" event={"ID":"4f991c4d-a12d-41b1-af5b-d806c2441ddf","Type":"ContainerStarted","Data":"6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267"} Oct 31 03:55:07.596170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:07.594206 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:55:08.288106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:08.288065 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:55:08.288739 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:08.288702 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:55:08.360341 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:08.360277 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" podStartSLOduration=9.360261982 podStartE2EDuration="9.360261982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:08.348904841 +0000 UTC m=+1605.349530071" watchObservedRunningTime="2026-10-31 03:55:08.360261982 +0000 UTC m=+1605.360887220" Oct 31 03:55:08.481330 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:08.481220 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:55:08.481330 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:08.481275 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 03:55:09.288859 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:09.288815 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:55:09.289274 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:09.288885 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:55:09.475437 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:09.475388 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:55:09.541825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:09.541730 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:55:09.621050 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:09.621015 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 03:55:10.287995 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:10.287959 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:55:10.499639 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:10.499604 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:55:10.639300 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:10.639207 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 03:55:13.276284 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.276242 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd"] Oct 31 03:55:13.281844 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.281817 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.383180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.382203 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.383180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.382277 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.383180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.382428 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9bsd\" (UniqueName: \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-kube-api-access-z9bsd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.383180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.382467 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.383180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.382525 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.383180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.382632 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.383180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.382825 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.385186 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.385154 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd"] Oct 31 03:55:13.408448 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.408415 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 03:55:13.410196 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.409438 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bfz2c\"" Oct 31 03:55:13.411327 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.409484 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 03:55:13.411327 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.409524 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 03:55:13.484904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.484811 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.485252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.485227 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.485443 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.485422 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.485615 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.485597 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9bsd\" (UniqueName: \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-kube-api-access-z9bsd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.485750 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.485729 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.485899 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.485882 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.486106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.486087 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.486427 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.486168 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.489493 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.489468 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.489735 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.489710 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.490772 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.490747 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.492850 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.492827 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.543761 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.543724 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.544100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.544074 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9bsd\" (UniqueName: \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-kube-api-access-z9bsd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kwcsd\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:13.600066 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:13.600032 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:14.179962 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:14.179917 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd"] Oct 31 03:55:14.217962 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:55:14.217931 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf825f3ae_e9e6_4358_aca1_9e575b0d6d91.slice/crio-ab913b04786952095b3bc7f4f92622f3c8387be0bfc664dc7bcada47265e274c WatchSource:0}: Error finding container ab913b04786952095b3bc7f4f92622f3c8387be0bfc664dc7bcada47265e274c: Status 404 returned error can't find the container with id ab913b04786952095b3bc7f4f92622f3c8387be0bfc664dc7bcada47265e274c Oct 31 03:55:14.743140 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:14.743085 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" event={"ID":"f825f3ae-e9e6-4358-aca1-9e575b0d6d91","Type":"ContainerStarted","Data":"ab913b04786952095b3bc7f4f92622f3c8387be0bfc664dc7bcada47265e274c"} Oct 31 03:55:18.611010 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:18.610868 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:55:18.737660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:18.737621 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:55:18.737831 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:18.737702 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:55:19.810665 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:19.810623 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" event={"ID":"f825f3ae-e9e6-4358-aca1-9e575b0d6d91","Type":"ContainerStarted","Data":"c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5"} Oct 31 03:55:19.811146 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:19.810746 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:19.817291 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:19.817263 4426 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kwcsd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:55:19.817408 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:19.817317 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" podUID="f825f3ae-e9e6-4358-aca1-9e575b0d6d91" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:55:20.009921 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:20.009826 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" podStartSLOduration=2.49380179 podStartE2EDuration="7.009803122s" totalImagesPullingTime="4.516001332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:19.972855785 +0000 UTC m=+1616.973481013" watchObservedRunningTime="2026-10-31 03:55:20.009803122 +0000 UTC m=+1617.010428351" Oct 31 03:55:20.822523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:20.822491 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 03:55:23.974653 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:23.974613 4426 scope.go:122] "RemoveContainer" containerID="030327485b876353644874d791a0787abdd7471366c1740323d717e4dd153a7c" Oct 31 03:55:23.987394 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:23.987369 4426 scope.go:122] "RemoveContainer" containerID="1dcec9433e6d743816dec677c899f18525b02362abcd6c1adce5b4d404988f75" Oct 31 03:55:23.998305 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:23.998281 4426 scope.go:122] "RemoveContainer" containerID="ef879b07ceac03a00eac604b1ea831c3374fd5f1bc07ed57e63bb4aa95634f52" Oct 31 03:55:24.009904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:24.009876 4426 scope.go:122] "RemoveContainer" containerID="e1ffa0e1818624d1ada84220029317cd1c7b909d021a36f3d49b13703c13d1d4" Oct 31 03:55:24.021952 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:24.021930 4426 scope.go:122] "RemoveContainer" containerID="dc19b07d4f302b577227a69c8056ac7bdb1305b00ca885a7db4347691e9b88dd" Oct 31 03:55:24.034357 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:24.034333 4426 scope.go:122] "RemoveContainer" containerID="f2c41ed583c8e96bff6b3397fbbdd8696dca7a977769872fd5f3635114785867" Oct 31 03:55:24.045549 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:24.045522 4426 scope.go:122] "RemoveContainer" containerID="cd2267adb2fb70f391c7c1d698d73b428d118f6e4d8a8e84525c47d7cbb2fab9" Oct 31 03:55:28.623333 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.623294 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9"] Oct 31 03:55:28.630517 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.630486 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:28.688783 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.688747 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9"] Oct 31 03:55:28.715237 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.714683 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d92ac050-bnkc9\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:28.715237 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.714749 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7kfx\" (UniqueName: \"kubernetes.io/projected/48b23084-662a-40f2-a816-e752ef0cfc16-kube-api-access-j7kfx\") pod \"maas-api-revoke-sub-d92ac050-bnkc9\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:28.715237 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.714816 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d92ac050-bnkc9\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:28.726691 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.726657 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:55:28.727072 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.727052 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:55:28.745495 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.745460 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:55:28.745794 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.745620 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:55:28.817025 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.816984 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d92ac050-bnkc9\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:28.817257 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.817045 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7kfx\" (UniqueName: \"kubernetes.io/projected/48b23084-662a-40f2-a816-e752ef0cfc16-kube-api-access-j7kfx\") pod \"maas-api-revoke-sub-d92ac050-bnkc9\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:28.817257 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.817098 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d92ac050-bnkc9\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:28.818097 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.818058 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d92ac050-bnkc9\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:28.819612 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.819577 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d92ac050-bnkc9\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:28.874893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.874814 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7kfx\" (UniqueName: \"kubernetes.io/projected/48b23084-662a-40f2-a816-e752ef0cfc16-kube-api-access-j7kfx\") pod \"maas-api-revoke-sub-d92ac050-bnkc9\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:28.949250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:28.949177 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:29.346531 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:29.346506 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9"] Oct 31 03:55:29.350044 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:55:29.350009 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b23084_662a_40f2_a816_e752ef0cfc16.slice/crio-da378835dc288701cdd8ac2eeef5c3c4aaf0cd308293bca3da4ac198598196d4 WatchSource:0}: Error finding container da378835dc288701cdd8ac2eeef5c3c4aaf0cd308293bca3da4ac198598196d4: Status 404 returned error can't find the container with id da378835dc288701cdd8ac2eeef5c3c4aaf0cd308293bca3da4ac198598196d4 Oct 31 03:55:29.947964 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:29.947912 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" event={"ID":"48b23084-662a-40f2-a816-e752ef0cfc16","Type":"ContainerStarted","Data":"5ed1198979af2539017324a79eb59bf9c663b50ea4d83a79b096cbe1f54f8c2f"} Oct 31 03:55:29.948451 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:29.947987 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" event={"ID":"48b23084-662a-40f2-a816-e752ef0cfc16","Type":"ContainerStarted","Data":"da378835dc288701cdd8ac2eeef5c3c4aaf0cd308293bca3da4ac198598196d4"} Oct 31 03:55:30.058371 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:30.058307 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" podStartSLOduration=2.058290894 podStartE2EDuration="2.058290894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:30.0567892 +0000 UTC m=+1627.057414431" watchObservedRunningTime="2026-10-31 03:55:30.058290894 +0000 UTC m=+1627.058916123" Oct 31 03:55:30.961814 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:30.961773 4426 generic.go:356] "Generic (PLEG): container finished" podID="48b23084-662a-40f2-a816-e752ef0cfc16" containerID="5ed1198979af2539017324a79eb59bf9c663b50ea4d83a79b096cbe1f54f8c2f" exitCode=0 Oct 31 03:55:30.962338 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:30.961850 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" event={"ID":"48b23084-662a-40f2-a816-e752ef0cfc16","Type":"ContainerDied","Data":"5ed1198979af2539017324a79eb59bf9c663b50ea4d83a79b096cbe1f54f8c2f"} Oct 31 03:55:31.742306 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:31.742264 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:55:32.164885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.164856 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:32.281206 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.279704 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-openshift-ca\") pod \"48b23084-662a-40f2-a816-e752ef0cfc16\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " Oct 31 03:55:32.281206 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.279839 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b23084-662a-40f2-a816-e752ef0cfc16-kube-api-access-j7kfx\" (UniqueName: \"kubernetes.io/projected/48b23084-662a-40f2-a816-e752ef0cfc16-kube-api-access-j7kfx\") pod \"48b23084-662a-40f2-a816-e752ef0cfc16\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " Oct 31 03:55:32.281206 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.280041 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-xks-ca\") pod \"48b23084-662a-40f2-a816-e752ef0cfc16\" (UID: \"48b23084-662a-40f2-a816-e752ef0cfc16\") " Oct 31 03:55:32.311472 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.287580 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-openshift-ca" pod "48b23084-662a-40f2-a816-e752ef0cfc16" (UID: "48b23084-662a-40f2-a816-e752ef0cfc16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:32.330363 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.330288 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-xks-ca" pod "48b23084-662a-40f2-a816-e752ef0cfc16" (UID: "48b23084-662a-40f2-a816-e752ef0cfc16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:32.405270 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.405210 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:32.405529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.405506 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b23084-662a-40f2-a816-e752ef0cfc16-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:32.421200 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.415612 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b23084-662a-40f2-a816-e752ef0cfc16-kube-api-access-j7kfx" pod "48b23084-662a-40f2-a816-e752ef0cfc16" (UID: "48b23084-662a-40f2-a816-e752ef0cfc16"). InnerVolumeSpecName "kube-api-access-j7kfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:32.571207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.517903 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7kfx\" (UniqueName: \"kubernetes.io/projected/48b23084-662a-40f2-a816-e752ef0cfc16-kube-api-access-j7kfx\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:32.903153 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:55:32.900925 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:55:32.988429 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.988396 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" event={"ID":"48b23084-662a-40f2-a816-e752ef0cfc16","Type":"ContainerDied","Data":"da378835dc288701cdd8ac2eeef5c3c4aaf0cd308293bca3da4ac198598196d4"} Oct 31 03:55:32.988731 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.988710 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da378835dc288701cdd8ac2eeef5c3c4aaf0cd308293bca3da4ac198598196d4" Oct 31 03:55:32.988957 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:32.988940 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9" Oct 31 03:55:34.694152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:34.693749 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:34.790272 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:34.790224 4426 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:54.325968 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.325930 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2"] Oct 31 03:55:54.328306 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.328278 4426 cpu_manager.go:414] "containerMap: removing container" podUID="48b23084-662a-40f2-a816-e752ef0cfc16" containerName="revoke-keys" Oct 31 03:55:54.328490 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.328476 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b23084-662a-40f2-a816-e752ef0cfc16" containerName="revoke-keys" Oct 31 03:55:54.329031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.329008 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b23084-662a-40f2-a816-e752ef0cfc16" containerName="revoke-keys" Oct 31 03:55:54.340725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.340695 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.389647 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.389614 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wk9x4\"" Oct 31 03:55:54.444704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.444459 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrs5\" (UniqueName: \"kubernetes.io/projected/8b99c850-aaaf-4055-9a35-dc3ce08085d7-kube-api-access-nqrs5\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.444704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.444621 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.444704 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.444664 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.445040 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.444732 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.445040 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.444778 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.445040 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.444810 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.445040 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.444838 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.445040 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.444918 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.445040 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.445007 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.472311 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.472274 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2"] Oct 31 03:55:54.547029 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.546985 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.547336 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.547097 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.547336 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.547232 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrs5\" (UniqueName: \"kubernetes.io/projected/8b99c850-aaaf-4055-9a35-dc3ce08085d7-kube-api-access-nqrs5\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.547466 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.547387 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.547466 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.547430 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.547575 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.547502 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.547575 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.547549 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.547686 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.547581 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.547686 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.547609 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.548141 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.548095 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.548617 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.548561 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.548829 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.548778 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.549657 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.549022 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.549657 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.549613 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.551377 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.551354 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.551584 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.551558 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.605425 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.605362 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b99c850-aaaf-4055-9a35-dc3ce08085d7-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.608908 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.608874 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrs5\" (UniqueName: \"kubernetes.io/projected/8b99c850-aaaf-4055-9a35-dc3ce08085d7-kube-api-access-nqrs5\") pod \"maas-default-gateway-openshift-default-7854949c9-tg5t2\" (UID: \"8b99c850-aaaf-4055-9a35-dc3ce08085d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:54.659169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:54.658722 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:55.168436 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:55:55.168398 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b99c850_aaaf_4055_9a35_dc3ce08085d7.slice/crio-8ed0c62f9c679b40ba4a9d2e0c0f5b5c9ef0162b4f7122ffe0eeaedbb8944104 WatchSource:0}: Error finding container 8ed0c62f9c679b40ba4a9d2e0c0f5b5c9ef0162b4f7122ffe0eeaedbb8944104: Status 404 returned error can't find the container with id 8ed0c62f9c679b40ba4a9d2e0c0f5b5c9ef0162b4f7122ffe0eeaedbb8944104 Oct 31 03:55:55.169590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:55.169560 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2"] Oct 31 03:55:55.171527 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:55.171488 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:55:55.171627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:55.171576 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:55:55.171627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:55.171608 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:55:55.491407 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:55.491315 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" event={"ID":"8b99c850-aaaf-4055-9a35-dc3ce08085d7","Type":"ContainerStarted","Data":"acc7d9353a2d2cc856edd1288b5921b912eccce02d1b1c12437018458b985048"} Oct 31 03:55:55.491407 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:55.491376 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" event={"ID":"8b99c850-aaaf-4055-9a35-dc3ce08085d7","Type":"ContainerStarted","Data":"8ed0c62f9c679b40ba4a9d2e0c0f5b5c9ef0162b4f7122ffe0eeaedbb8944104"} Oct 31 03:55:55.605109 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:55.605051 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" podStartSLOduration=1.6050357480000002 podStartE2EDuration="1.605035748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:55.603743526 +0000 UTC m=+1652.604368757" watchObservedRunningTime="2026-10-31 03:55:55.605035748 +0000 UTC m=+1652.605660976" Oct 31 03:55:55.659250 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:55.659213 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:55:56.662312 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:56.661252 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:55:56.662312 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:56.661312 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" podUID="8b99c850-aaaf-4055-9a35-dc3ce08085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:55:57.661374 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:57.661329 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:55:57.661599 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:57.661409 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" podUID="8b99c850-aaaf-4055-9a35-dc3ce08085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:55:58.661208 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:58.659293 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:55:58.661208 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:58.659377 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" podUID="8b99c850-aaaf-4055-9a35-dc3ce08085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:55:59.659264 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:59.659229 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:55:59.659489 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:55:59.659303 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" podUID="8b99c850-aaaf-4055-9a35-dc3ce08085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:56:00.659110 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:00.659073 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:56:00.659740 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:00.659176 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" podUID="8b99c850-aaaf-4055-9a35-dc3ce08085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:56:01.659575 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:01.659520 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:56:01.660215 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:01.660172 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" podUID="8b99c850-aaaf-4055-9a35-dc3ce08085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:56:02.662601 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:02.662556 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:56:02.663287 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:02.663247 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" podUID="8b99c850-aaaf-4055-9a35-dc3ce08085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:56:02.849659 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:56:02.849614 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:56:03.659061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:03.659027 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:56:03.659391 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:03.659090 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" podUID="8b99c850-aaaf-4055-9a35-dc3ce08085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:56:04.778156 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:04.778109 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:56:04.778980 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:04.778953 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:56:04.780022 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:04.779996 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" Oct 31 03:56:09.433176 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.433073 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz"] Oct 31 03:56:09.438468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.438440 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.542290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.542260 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz"] Oct 31 03:56:09.567291 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.567198 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.567523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.567337 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.567523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.567384 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d600e53-d628-4f89-b6c6-186d3432e300-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.567523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.567421 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.567523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.567447 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.567751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.567570 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.567751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.567603 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fs5n\" (UniqueName: \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-kube-api-access-4fs5n\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.567751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.567657 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d600e53-d628-4f89-b6c6-186d3432e300-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.567751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.567744 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.669238 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.669189 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d600e53-d628-4f89-b6c6-186d3432e300-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.669451 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.669291 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.669451 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.669410 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.669580 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.669522 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.669580 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.669566 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d600e53-d628-4f89-b6c6-186d3432e300-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.669687 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.669605 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.669687 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.669632 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.669797 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.669773 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.669852 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.669806 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fs5n\" (UniqueName: \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-kube-api-access-4fs5n\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.669909 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.669870 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.670100 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.670065 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.670264 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.670176 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d600e53-d628-4f89-b6c6-186d3432e300-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.670328 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.670298 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.670385 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.670365 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.672664 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.672638 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.672833 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.672809 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d600e53-d628-4f89-b6c6-186d3432e300-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.727345 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.727262 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.727517 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.727394 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fs5n\" (UniqueName: \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-kube-api-access-4fs5n\") pod \"maas-default-gateway-openshift-default-7854949c9-fp9bz\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:09.755467 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:09.755434 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:10.214999 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:56:10.214950 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d600e53_d628_4f89_b6c6_186d3432e300.slice/crio-61e2ed21683b224bd594bbbd60c9a2cfee2d8656a4393441857b7ee0747f2b4d WatchSource:0}: Error finding container 61e2ed21683b224bd594bbbd60c9a2cfee2d8656a4393441857b7ee0747f2b4d: Status 404 returned error can't find the container with id 61e2ed21683b224bd594bbbd60c9a2cfee2d8656a4393441857b7ee0747f2b4d Oct 31 03:56:10.217038 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:10.217008 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz"] Oct 31 03:56:10.218035 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:10.217985 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:56:10.218186 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:10.218073 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:56:10.218186 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:10.218118 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:56:10.718699 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:10.718656 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" event={"ID":"4d600e53-d628-4f89-b6c6-186d3432e300","Type":"ContainerStarted","Data":"990d7b5af68d24b3bd3c7a4f29dc0182e33e6e9c8a6b8247d043d06ef976216c"} Oct 31 03:56:10.719226 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:10.718726 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" event={"ID":"4d600e53-d628-4f89-b6c6-186d3432e300","Type":"ContainerStarted","Data":"61e2ed21683b224bd594bbbd60c9a2cfee2d8656a4393441857b7ee0747f2b4d"} Oct 31 03:56:10.756511 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:10.756470 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:10.842579 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:10.840960 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" podStartSLOduration=1.840935509 podStartE2EDuration="1.840935509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:10.833802886 +0000 UTC m=+1667.834428114" watchObservedRunningTime="2026-10-31 03:56:10.840935509 +0000 UTC m=+1667.841560738" Oct 31 03:56:11.757705 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:11.757668 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fp9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:56:11.758108 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:11.757731 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:56:12.055443 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:12.055353 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd"] Oct 31 03:56:12.098061 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:12.098022 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49e9114-qhwjd"] Oct 31 03:56:12.756120 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:12.756079 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fp9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:56:12.756354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:12.756175 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:56:13.159008 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:13.158973 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca520e83-5d3e-4293-9312-de39429cd6fc" path="/var/lib/kubelet/pods/ca520e83-5d3e-4293-9312-de39429cd6fc/volumes" Oct 31 03:56:13.756391 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:13.756356 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fp9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:56:13.756607 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:13.756412 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:56:14.756656 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:14.756621 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fp9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:56:14.757073 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:14.756688 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:56:15.756105 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:15.756069 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fp9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:56:15.756326 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:15.756155 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:56:16.756290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:16.756250 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fp9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:56:16.756666 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:16.756354 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:56:17.756861 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:17.756820 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fp9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:56:17.757365 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:17.756906 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:56:19.146084 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:19.146046 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:19.147237 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:19.147211 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:19.148041 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:19.148018 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 03:56:32.851656 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:56:32.851610 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:56:35.396037 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:35.395992 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:56:38.627868 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.627835 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx"] Oct 31 03:56:38.634343 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.634324 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:38.687530 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.687497 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx"] Oct 31 03:56:38.688329 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.688302 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:38.688435 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.688381 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4czx\" (UniqueName: \"kubernetes.io/projected/629e1d07-6469-4a4d-ae65-786956ebf160-kube-api-access-n4czx\") pod \"maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:38.688623 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.688609 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:38.726685 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.726645 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:56:38.726685 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.726672 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:56:38.729652 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.729623 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:56:38.729652 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.729631 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:56:38.790071 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.790034 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:38.790294 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.790104 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:38.790294 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.790199 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4czx\" (UniqueName: \"kubernetes.io/projected/629e1d07-6469-4a4d-ae65-786956ebf160-kube-api-access-n4czx\") pod \"maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:38.790772 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.790748 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:38.792598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.792577 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:38.848021 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.847987 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4czx\" (UniqueName: \"kubernetes.io/projected/629e1d07-6469-4a4d-ae65-786956ebf160-kube-api-access-n4czx\") pod \"maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:38.949332 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:38.949295 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:39.342893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:39.342848 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx"] Oct 31 03:56:39.344849 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:56:39.344807 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629e1d07_6469_4a4d_ae65_786956ebf160.slice/crio-454efedc8391f7761687dc443c62c95dc8a5e96fba1a810c4f2516c19c9df4d2 WatchSource:0}: Error finding container 454efedc8391f7761687dc443c62c95dc8a5e96fba1a810c4f2516c19c9df4d2: Status 404 returned error can't find the container with id 454efedc8391f7761687dc443c62c95dc8a5e96fba1a810c4f2516c19c9df4d2 Oct 31 03:56:40.074627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:40.074528 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" event={"ID":"629e1d07-6469-4a4d-ae65-786956ebf160","Type":"ContainerStarted","Data":"42ca271eda326be0e1a33ca86798d4f11122cbd9cbd9a669aee8e4cab7b740df"} Oct 31 03:56:40.074627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:40.074593 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" event={"ID":"629e1d07-6469-4a4d-ae65-786956ebf160","Type":"ContainerStarted","Data":"454efedc8391f7761687dc443c62c95dc8a5e96fba1a810c4f2516c19c9df4d2"} Oct 31 03:56:40.185114 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:40.185056 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" podStartSLOduration=2.185042329 podStartE2EDuration="2.185042329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:40.183385519 +0000 UTC m=+1697.184010746" watchObservedRunningTime="2026-10-31 03:56:40.185042329 +0000 UTC m=+1697.185667556" Oct 31 03:56:41.086850 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:41.086813 4426 generic.go:356] "Generic (PLEG): container finished" podID="629e1d07-6469-4a4d-ae65-786956ebf160" containerID="42ca271eda326be0e1a33ca86798d4f11122cbd9cbd9a669aee8e4cab7b740df" exitCode=0 Oct 31 03:56:41.087289 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:41.086892 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" event={"ID":"629e1d07-6469-4a4d-ae65-786956ebf160","Type":"ContainerDied","Data":"42ca271eda326be0e1a33ca86798d4f11122cbd9cbd9a669aee8e4cab7b740df"} Oct 31 03:56:42.328081 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:42.328048 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:42.453455 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:42.453423 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-openshift-ca\") pod \"629e1d07-6469-4a4d-ae65-786956ebf160\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " Oct 31 03:56:42.453650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:42.453525 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-xks-ca\") pod \"629e1d07-6469-4a4d-ae65-786956ebf160\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " Oct 31 03:56:42.453650 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:42.453644 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629e1d07-6469-4a4d-ae65-786956ebf160-kube-api-access-n4czx\" (UniqueName: \"kubernetes.io/projected/629e1d07-6469-4a4d-ae65-786956ebf160-kube-api-access-n4czx\") pod \"629e1d07-6469-4a4d-ae65-786956ebf160\" (UID: \"629e1d07-6469-4a4d-ae65-786956ebf160\") " Oct 31 03:56:42.453843 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:42.453816 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-openshift-ca" pod "629e1d07-6469-4a4d-ae65-786956ebf160" (UID: "629e1d07-6469-4a4d-ae65-786956ebf160"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:42.454285 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:42.454260 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:42.456560 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:42.456529 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629e1d07-6469-4a4d-ae65-786956ebf160-kube-api-access-n4czx" pod "629e1d07-6469-4a4d-ae65-786956ebf160" (UID: "629e1d07-6469-4a4d-ae65-786956ebf160"). InnerVolumeSpecName "kube-api-access-n4czx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:42.456683 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:42.456649 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-xks-ca" pod "629e1d07-6469-4a4d-ae65-786956ebf160" (UID: "629e1d07-6469-4a4d-ae65-786956ebf160"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:42.555819 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:42.555783 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629e1d07-6469-4a4d-ae65-786956ebf160-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:42.555819 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:42.555817 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4czx\" (UniqueName: \"kubernetes.io/projected/629e1d07-6469-4a4d-ae65-786956ebf160-kube-api-access-n4czx\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:43.129271 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:43.129238 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" Oct 31 03:56:43.133497 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:43.133467 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx" event={"ID":"629e1d07-6469-4a4d-ae65-786956ebf160","Type":"ContainerDied","Data":"454efedc8391f7761687dc443c62c95dc8a5e96fba1a810c4f2516c19c9df4d2"} Oct 31 03:56:43.133667 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:43.133514 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="454efedc8391f7761687dc443c62c95dc8a5e96fba1a810c4f2516c19c9df4d2" Oct 31 03:56:48.154232 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.154192 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl"] Oct 31 03:56:48.154998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.154980 4426 cpu_manager.go:414] "containerMap: removing container" podUID="629e1d07-6469-4a4d-ae65-786956ebf160" containerName="revoke-keys" Oct 31 03:56:48.154998 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.154998 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629e1d07-6469-4a4d-ae65-786956ebf160" containerName="revoke-keys" Oct 31 03:56:48.155175 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.155164 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="629e1d07-6469-4a4d-ae65-786956ebf160" containerName="revoke-keys" Oct 31 03:56:48.159579 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.159555 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:48.207212 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.207180 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl"] Oct 31 03:56:48.246468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.246436 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:56:48.246707 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.246448 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:56:48.255029 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.254991 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:56:48.255214 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.255006 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:56:48.259478 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.259445 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:48.260355 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.260327 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:48.260660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.260638 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fsr7\" (UniqueName: \"kubernetes.io/projected/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-kube-api-access-9fsr7\") pod \"maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:48.361956 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.361916 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:48.362171 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.362046 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:48.362249 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.362170 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fsr7\" (UniqueName: \"kubernetes.io/projected/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-kube-api-access-9fsr7\") pod \"maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:48.362972 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.362942 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:48.364601 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.364576 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:48.420930 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.420837 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fsr7\" (UniqueName: \"kubernetes.io/projected/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-kube-api-access-9fsr7\") pod \"maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:48.474660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.474622 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:48.874933 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:48.874904 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl"] Oct 31 03:56:48.877357 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:56:48.877330 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cb536a_dfcc_4ca4_ac19_0e222b8bd789.slice/crio-23a42292e21a8f7af98b404ec24a3d0fe55bd3bfc7990bd06c1c5bfdf1415862 WatchSource:0}: Error finding container 23a42292e21a8f7af98b404ec24a3d0fe55bd3bfc7990bd06c1c5bfdf1415862: Status 404 returned error can't find the container with id 23a42292e21a8f7af98b404ec24a3d0fe55bd3bfc7990bd06c1c5bfdf1415862 Oct 31 03:56:49.275070 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:49.275029 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" event={"ID":"23cb536a-dfcc-4ca4-ac19-0e222b8bd789","Type":"ContainerStarted","Data":"23a42292e21a8f7af98b404ec24a3d0fe55bd3bfc7990bd06c1c5bfdf1415862"} Oct 31 03:56:49.314709 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:49.314662 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf"] Oct 31 03:56:50.293649 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:50.293613 4426 generic.go:356] "Generic (PLEG): container finished" podID="23cb536a-dfcc-4ca4-ac19-0e222b8bd789" containerID="f37f7c90f8e66159f1d4322baa37f936319cfe3178443cd4873c68c35e956653" exitCode=0 Oct 31 03:56:50.294088 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:50.293805 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" event={"ID":"23cb536a-dfcc-4ca4-ac19-0e222b8bd789","Type":"ContainerDied","Data":"f37f7c90f8e66159f1d4322baa37f936319cfe3178443cd4873c68c35e956653"} Oct 31 03:56:51.536957 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:51.536920 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:51.734439 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:51.734398 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-kube-api-access-9fsr7\" (UniqueName: \"kubernetes.io/projected/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-kube-api-access-9fsr7\") pod \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " Oct 31 03:56:51.734715 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:51.734695 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-openshift-ca\") pod \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " Oct 31 03:56:51.734805 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:51.734793 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-xks-ca\") pod \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\" (UID: \"23cb536a-dfcc-4ca4-ac19-0e222b8bd789\") " Oct 31 03:56:51.735180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:51.735154 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-openshift-ca" pod "23cb536a-dfcc-4ca4-ac19-0e222b8bd789" (UID: "23cb536a-dfcc-4ca4-ac19-0e222b8bd789"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:51.735545 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:51.735525 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.736897 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:51.736866 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-xks-ca" pod "23cb536a-dfcc-4ca4-ac19-0e222b8bd789" (UID: "23cb536a-dfcc-4ca4-ac19-0e222b8bd789"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:51.737114 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:51.737087 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-kube-api-access-9fsr7" pod "23cb536a-dfcc-4ca4-ac19-0e222b8bd789" (UID: "23cb536a-dfcc-4ca4-ac19-0e222b8bd789"). InnerVolumeSpecName "kube-api-access-9fsr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:51.837405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:51.837362 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.837405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:51.837402 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fsr7\" (UniqueName: \"kubernetes.io/projected/23cb536a-dfcc-4ca4-ac19-0e222b8bd789-kube-api-access-9fsr7\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.315914 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.315881 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" Oct 31 03:56:52.315914 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.315905 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl" event={"ID":"23cb536a-dfcc-4ca4-ac19-0e222b8bd789","Type":"ContainerDied","Data":"23a42292e21a8f7af98b404ec24a3d0fe55bd3bfc7990bd06c1c5bfdf1415862"} Oct 31 03:56:52.316167 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.315983 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23a42292e21a8f7af98b404ec24a3d0fe55bd3bfc7990bd06c1c5bfdf1415862" Oct 31 03:56:52.888586 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.888547 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq"] Oct 31 03:56:52.889706 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.889671 4426 cpu_manager.go:414] "containerMap: removing container" podUID="23cb536a-dfcc-4ca4-ac19-0e222b8bd789" containerName="revoke-keys" Oct 31 03:56:52.889706 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.889700 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cb536a-dfcc-4ca4-ac19-0e222b8bd789" containerName="revoke-keys" Oct 31 03:56:52.890023 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.890001 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cb536a-dfcc-4ca4-ac19-0e222b8bd789" containerName="revoke-keys" Oct 31 03:56:52.898698 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.895913 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:52.949967 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.946581 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:56:52.954749 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.954715 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq"] Oct 31 03:56:52.959329 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.959286 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:52.959453 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.959422 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnmx\" (UniqueName: \"kubernetes.io/projected/3b756361-767b-433e-b14d-db11a3c3dd95-kube-api-access-gsnmx\") pod \"maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:52.959574 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.959557 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:52.984281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.984243 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:56:52.984459 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.984384 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:56:52.984459 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:52.984437 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:56:53.060825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:53.060786 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnmx\" (UniqueName: \"kubernetes.io/projected/3b756361-767b-433e-b14d-db11a3c3dd95-kube-api-access-gsnmx\") pod \"maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:53.061034 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:53.060935 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:53.061194 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:53.061173 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:53.061613 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:53.061588 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:53.063468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:53.063447 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:53.121609 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:53.121580 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnmx\" (UniqueName: \"kubernetes.io/projected/3b756361-767b-433e-b14d-db11a3c3dd95-kube-api-access-gsnmx\") pod \"maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:53.221952 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:53.221909 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:53.643106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:53.643071 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq"] Oct 31 03:56:53.643847 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:56:53.643799 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b756361_767b_433e_b14d_db11a3c3dd95.slice/crio-4871b372c4c0a5e7bbe4a48e435f1778352a2ca8177529a960f40f7d19165c9a WatchSource:0}: Error finding container 4871b372c4c0a5e7bbe4a48e435f1778352a2ca8177529a960f40f7d19165c9a: Status 404 returned error can't find the container with id 4871b372c4c0a5e7bbe4a48e435f1778352a2ca8177529a960f40f7d19165c9a Oct 31 03:56:54.315732 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:54.315625 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" podUID="4f991c4d-a12d-41b1-af5b-d806c2441ddf" containerName="payload-pre-processing" containerID="cri-o://6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267" gracePeriod=25 Oct 31 03:56:54.346442 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:54.346403 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" event={"ID":"3b756361-767b-433e-b14d-db11a3c3dd95","Type":"ContainerStarted","Data":"62754e8d71090d161e0f8d75ca4824057e0968de6baee2f8882734a5f17c20d7"} Oct 31 03:56:54.346587 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:54.346473 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" event={"ID":"3b756361-767b-433e-b14d-db11a3c3dd95","Type":"ContainerStarted","Data":"4871b372c4c0a5e7bbe4a48e435f1778352a2ca8177529a960f40f7d19165c9a"} Oct 31 03:56:54.464642 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:54.464569 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" podStartSLOduration=2.464549277 podStartE2EDuration="2.464549277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:54.456094398 +0000 UTC m=+1711.456719629" watchObservedRunningTime="2026-10-31 03:56:54.464549277 +0000 UTC m=+1711.465174506" Oct 31 03:56:54.667740 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:54.667708 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:56:54.786634 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:54.786599 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f991c4d-a12d-41b1-af5b-d806c2441ddf-kube-api-access-lzw85\" (UniqueName: \"kubernetes.io/projected/4f991c4d-a12d-41b1-af5b-d806c2441ddf-kube-api-access-lzw85\") pod \"4f991c4d-a12d-41b1-af5b-d806c2441ddf\" (UID: \"4f991c4d-a12d-41b1-af5b-d806c2441ddf\") " Oct 31 03:56:54.786813 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:54.786744 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f991c4d-a12d-41b1-af5b-d806c2441ddf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f991c4d-a12d-41b1-af5b-d806c2441ddf-plugins-config-volume\") pod \"4f991c4d-a12d-41b1-af5b-d806c2441ddf\" (UID: \"4f991c4d-a12d-41b1-af5b-d806c2441ddf\") " Oct 31 03:56:54.787103 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:54.787079 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f991c4d-a12d-41b1-af5b-d806c2441ddf-plugins-config-volume" pod "4f991c4d-a12d-41b1-af5b-d806c2441ddf" (UID: "4f991c4d-a12d-41b1-af5b-d806c2441ddf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:54.787405 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:54.787386 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f991c4d-a12d-41b1-af5b-d806c2441ddf-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:54.788795 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:54.788762 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f991c4d-a12d-41b1-af5b-d806c2441ddf-kube-api-access-lzw85" pod "4f991c4d-a12d-41b1-af5b-d806c2441ddf" (UID: "4f991c4d-a12d-41b1-af5b-d806c2441ddf"). InnerVolumeSpecName "kube-api-access-lzw85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:54.888804 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:54.888707 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzw85\" (UniqueName: \"kubernetes.io/projected/4f991c4d-a12d-41b1-af5b-d806c2441ddf-kube-api-access-lzw85\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:55.364582 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:55.364537 4426 generic.go:356] "Generic (PLEG): container finished" podID="4f991c4d-a12d-41b1-af5b-d806c2441ddf" containerID="6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267" exitCode=0 Oct 31 03:56:55.365106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:55.364612 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" Oct 31 03:56:55.365106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:55.364638 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" event={"ID":"4f991c4d-a12d-41b1-af5b-d806c2441ddf","Type":"ContainerDied","Data":"6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267"} Oct 31 03:56:55.365106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:55.364773 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf" event={"ID":"4f991c4d-a12d-41b1-af5b-d806c2441ddf","Type":"ContainerDied","Data":"c94a124466850e1f368d0940db912ebde31437d3cce30f024440b3d4fad8b622"} Oct 31 03:56:55.365106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:55.364814 4426 scope.go:122] "RemoveContainer" containerID="6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267" Oct 31 03:56:55.369086 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:55.369051 4426 generic.go:356] "Generic (PLEG): container finished" podID="3b756361-767b-433e-b14d-db11a3c3dd95" containerID="62754e8d71090d161e0f8d75ca4824057e0968de6baee2f8882734a5f17c20d7" exitCode=0 Oct 31 03:56:55.369211 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:55.369103 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" event={"ID":"3b756361-767b-433e-b14d-db11a3c3dd95","Type":"ContainerDied","Data":"62754e8d71090d161e0f8d75ca4824057e0968de6baee2f8882734a5f17c20d7"} Oct 31 03:56:55.379495 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:55.379471 4426 scope.go:122] "RemoveContainer" containerID="6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267" Oct 31 03:56:55.379827 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:56:55.379796 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267\": container with ID starting with 6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267 not found: ID does not exist" containerID="6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267" Oct 31 03:56:55.379920 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:55.379838 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267"} err="failed to get container status \"6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267\": rpc error: code = NotFound desc = could not find container \"6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267\": container with ID starting with 6892d8853d74432c0ca964da9c4d9c28fa3f04c1156421faff0b9d740b66f267 not found: ID does not exist" Oct 31 03:56:55.634667 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:55.634574 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf"] Oct 31 03:56:55.640872 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:55.640843 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-34a72a1f-5dc9c9dfbc-q48zf"] Oct 31 03:56:56.592219 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:56.592191 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:56.719819 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:56.719780 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b756361-767b-433e-b14d-db11a3c3dd95-kube-api-access-gsnmx\" (UniqueName: \"kubernetes.io/projected/3b756361-767b-433e-b14d-db11a3c3dd95-kube-api-access-gsnmx\") pod \"3b756361-767b-433e-b14d-db11a3c3dd95\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " Oct 31 03:56:56.720035 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:56.720019 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-xks-ca\") pod \"3b756361-767b-433e-b14d-db11a3c3dd95\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " Oct 31 03:56:56.720137 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:56.720094 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-openshift-ca\") pod \"3b756361-767b-433e-b14d-db11a3c3dd95\" (UID: \"3b756361-767b-433e-b14d-db11a3c3dd95\") " Oct 31 03:56:56.720569 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:56.720534 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-openshift-ca" pod "3b756361-767b-433e-b14d-db11a3c3dd95" (UID: "3b756361-767b-433e-b14d-db11a3c3dd95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:56.721071 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:56.721040 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.722517 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:56.722493 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b756361-767b-433e-b14d-db11a3c3dd95-kube-api-access-gsnmx" pod "3b756361-767b-433e-b14d-db11a3c3dd95" (UID: "3b756361-767b-433e-b14d-db11a3c3dd95"). InnerVolumeSpecName "kube-api-access-gsnmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:56.722760 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:56.722739 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-xks-ca" pod "3b756361-767b-433e-b14d-db11a3c3dd95" (UID: "3b756361-767b-433e-b14d-db11a3c3dd95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:56.823168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:56.823114 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsnmx\" (UniqueName: \"kubernetes.io/projected/3b756361-767b-433e-b14d-db11a3c3dd95-kube-api-access-gsnmx\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.823168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:56.823167 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b756361-767b-433e-b14d-db11a3c3dd95-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:57.132358 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:57.132284 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f991c4d-a12d-41b1-af5b-d806c2441ddf" path="/var/lib/kubelet/pods/4f991c4d-a12d-41b1-af5b-d806c2441ddf/volumes" Oct 31 03:56:57.391450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:57.391416 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" Oct 31 03:56:57.391450 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:57.391434 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq" event={"ID":"3b756361-767b-433e-b14d-db11a3c3dd95","Type":"ContainerDied","Data":"4871b372c4c0a5e7bbe4a48e435f1778352a2ca8177529a960f40f7d19165c9a"} Oct 31 03:56:57.391665 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:56:57.391497 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4871b372c4c0a5e7bbe4a48e435f1778352a2ca8177529a960f40f7d19165c9a" Oct 31 03:57:02.848931 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:57:02.848885 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:57:04.053265 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:04.053227 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc"] Oct 31 03:57:04.099943 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:04.099910 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2da14249-xvcdc"] Oct 31 03:57:05.132237 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:05.132201 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5257abaf-67a0-45f6-80d2-077bc7e506d3" path="/var/lib/kubelet/pods/5257abaf-67a0-45f6-80d2-077bc7e506d3/volumes" Oct 31 03:57:10.090850 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.090811 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996"] Oct 31 03:57:10.091774 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.091754 4426 cpu_manager.go:414] "containerMap: removing container" podUID="4f991c4d-a12d-41b1-af5b-d806c2441ddf" containerName="payload-pre-processing" Oct 31 03:57:10.091824 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.091775 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f991c4d-a12d-41b1-af5b-d806c2441ddf" containerName="payload-pre-processing" Oct 31 03:57:10.091824 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.091785 4426 cpu_manager.go:414] "containerMap: removing container" podUID="3b756361-767b-433e-b14d-db11a3c3dd95" containerName="revoke-keys" Oct 31 03:57:10.091824 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.091790 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b756361-767b-433e-b14d-db11a3c3dd95" containerName="revoke-keys" Oct 31 03:57:10.091954 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.091944 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f991c4d-a12d-41b1-af5b-d806c2441ddf" containerName="payload-pre-processing" Oct 31 03:57:10.091992 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.091955 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b756361-767b-433e-b14d-db11a3c3dd95" containerName="revoke-keys" Oct 31 03:57:10.097517 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.097497 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:10.150810 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.150770 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996"] Oct 31 03:57:10.189045 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.189004 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:57:10.189045 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.189025 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:57:10.189340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.189097 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:57:10.189340 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.189108 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:57:10.194868 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.194839 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51ffba08-jc996\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:10.195028 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.194877 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5gc\" (UniqueName: \"kubernetes.io/projected/f845d43a-7e97-4986-acf6-cc3524d1b99f-kube-api-access-vb5gc\") pod \"maas-api-revoke-sub-51ffba08-jc996\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:10.195028 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.194999 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51ffba08-jc996\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:10.296398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.296366 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51ffba08-jc996\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:10.296561 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.296407 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5gc\" (UniqueName: \"kubernetes.io/projected/f845d43a-7e97-4986-acf6-cc3524d1b99f-kube-api-access-vb5gc\") pod \"maas-api-revoke-sub-51ffba08-jc996\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:10.296561 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.296464 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51ffba08-jc996\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:10.297162 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.297099 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51ffba08-jc996\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:10.298807 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.298783 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51ffba08-jc996\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:10.354550 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.354468 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5gc\" (UniqueName: \"kubernetes.io/projected/f845d43a-7e97-4986-acf6-cc3524d1b99f-kube-api-access-vb5gc\") pod \"maas-api-revoke-sub-51ffba08-jc996\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:10.412775 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.412727 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:10.819766 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:10.819733 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996"] Oct 31 03:57:10.821538 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:57:10.821510 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf845d43a_7e97_4986_acf6_cc3524d1b99f.slice/crio-025ae056634c6d826a3f56f89bd5b53aafd1b325238b67fff950f6e4e3d22b99 WatchSource:0}: Error finding container 025ae056634c6d826a3f56f89bd5b53aafd1b325238b67fff950f6e4e3d22b99: Status 404 returned error can't find the container with id 025ae056634c6d826a3f56f89bd5b53aafd1b325238b67fff950f6e4e3d22b99 Oct 31 03:57:11.562596 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:11.562552 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" event={"ID":"f845d43a-7e97-4986-acf6-cc3524d1b99f","Type":"ContainerStarted","Data":"f06c7c0f80182b1f8743e008d78fa19999b0d6fb82cd4c946e664c011a48b7d8"} Oct 31 03:57:11.563093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:11.562618 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" event={"ID":"f845d43a-7e97-4986-acf6-cc3524d1b99f","Type":"ContainerStarted","Data":"025ae056634c6d826a3f56f89bd5b53aafd1b325238b67fff950f6e4e3d22b99"} Oct 31 03:57:11.676404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:11.676348 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" podStartSLOduration=1.676335043 podStartE2EDuration="1.676335043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:57:11.670596538 +0000 UTC m=+1728.671221767" watchObservedRunningTime="2026-10-31 03:57:11.676335043 +0000 UTC m=+1728.676960271" Oct 31 03:57:12.574036 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:12.574002 4426 generic.go:356] "Generic (PLEG): container finished" podID="f845d43a-7e97-4986-acf6-cc3524d1b99f" containerID="f06c7c0f80182b1f8743e008d78fa19999b0d6fb82cd4c946e664c011a48b7d8" exitCode=0 Oct 31 03:57:12.574498 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:12.574098 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" event={"ID":"f845d43a-7e97-4986-acf6-cc3524d1b99f","Type":"ContainerDied","Data":"f06c7c0f80182b1f8743e008d78fa19999b0d6fb82cd4c946e664c011a48b7d8"} Oct 31 03:57:13.766536 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:13.766508 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:13.855281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:13.855185 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-openshift-ca\") pod \"f845d43a-7e97-4986-acf6-cc3524d1b99f\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " Oct 31 03:57:13.855474 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:13.855359 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f845d43a-7e97-4986-acf6-cc3524d1b99f-kube-api-access-vb5gc\" (UniqueName: \"kubernetes.io/projected/f845d43a-7e97-4986-acf6-cc3524d1b99f-kube-api-access-vb5gc\") pod \"f845d43a-7e97-4986-acf6-cc3524d1b99f\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " Oct 31 03:57:13.855474 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:13.855450 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-xks-ca\") pod \"f845d43a-7e97-4986-acf6-cc3524d1b99f\" (UID: \"f845d43a-7e97-4986-acf6-cc3524d1b99f\") " Oct 31 03:57:13.855594 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:13.855567 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-openshift-ca" pod "f845d43a-7e97-4986-acf6-cc3524d1b99f" (UID: "f845d43a-7e97-4986-acf6-cc3524d1b99f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:13.856290 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:13.856261 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:13.857612 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:13.857581 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-xks-ca" pod "f845d43a-7e97-4986-acf6-cc3524d1b99f" (UID: "f845d43a-7e97-4986-acf6-cc3524d1b99f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:13.857900 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:13.857873 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f845d43a-7e97-4986-acf6-cc3524d1b99f-kube-api-access-vb5gc" pod "f845d43a-7e97-4986-acf6-cc3524d1b99f" (UID: "f845d43a-7e97-4986-acf6-cc3524d1b99f"). InnerVolumeSpecName "kube-api-access-vb5gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:13.958329 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:13.958294 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb5gc\" (UniqueName: \"kubernetes.io/projected/f845d43a-7e97-4986-acf6-cc3524d1b99f-kube-api-access-vb5gc\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:13.958329 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:13.958325 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f845d43a-7e97-4986-acf6-cc3524d1b99f-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:14.593757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:14.593721 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" Oct 31 03:57:14.593757 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:14.593742 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996" event={"ID":"f845d43a-7e97-4986-acf6-cc3524d1b99f","Type":"ContainerDied","Data":"025ae056634c6d826a3f56f89bd5b53aafd1b325238b67fff950f6e4e3d22b99"} Oct 31 03:57:14.594013 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:14.593801 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="025ae056634c6d826a3f56f89bd5b53aafd1b325238b67fff950f6e4e3d22b99" Oct 31 03:57:19.049801 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:19.049764 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2"] Oct 31 03:57:19.099863 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:19.099815 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7b3a17-rcwb2"] Oct 31 03:57:19.132548 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:19.132516 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711e32e7-8af5-4e8d-83f2-221c67c3b55f" path="/var/lib/kubelet/pods/711e32e7-8af5-4e8d-83f2-221c67c3b55f/volumes" Oct 31 03:57:21.443840 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:21.443803 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx"] Oct 31 03:57:21.489855 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:21.489808 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-34a72a1f-67brx"] Oct 31 03:57:23.133653 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:23.133622 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629e1d07-6469-4a4d-ae65-786956ebf160" path="/var/lib/kubelet/pods/629e1d07-6469-4a4d-ae65-786956ebf160/volumes" Oct 31 03:57:24.137687 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:24.137654 4426 scope.go:122] "RemoveContainer" containerID="aedbcada780fb8ef42b4d8a8615bc26a05508c1847e69566d7f9213e2949c703" Oct 31 03:57:27.239514 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:27.239459 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl"] Oct 31 03:57:27.269612 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:27.269575 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-984fb1f2-5hfzl"] Oct 31 03:57:28.331063 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:28.331023 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq"] Oct 31 03:57:28.370777 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:28.370743 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af5cff2a-5qflq"] Oct 31 03:57:29.132590 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:29.132556 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23cb536a-dfcc-4ca4-ac19-0e222b8bd789" path="/var/lib/kubelet/pods/23cb536a-dfcc-4ca4-ac19-0e222b8bd789/volumes" Oct 31 03:57:29.133077 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:29.133062 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b756361-767b-433e-b14d-db11a3c3dd95" path="/var/lib/kubelet/pods/3b756361-767b-433e-b14d-db11a3c3dd95/volumes" Oct 31 03:57:33.136387 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:57:33.134057 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:57:37.391093 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:37.390987 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:57:38.068679 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:38.068639 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm"] Oct 31 03:57:38.097637 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:38.097594 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc1c4c3-thjgm"] Oct 31 03:57:39.134232 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:39.134198 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6eceb4e-b68e-4893-8879-60fedaf397b9" path="/var/lib/kubelet/pods/f6eceb4e-b68e-4893-8879-60fedaf397b9/volumes" Oct 31 03:57:45.066946 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:45.066906 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf"] Oct 31 03:57:45.099989 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:45.099957 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91678f80-t5bwf"] Oct 31 03:57:45.134846 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:45.134806 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1592df3f-7096-4d96-85af-0ba30ff950b2" path="/var/lib/kubelet/pods/1592df3f-7096-4d96-85af-0ba30ff950b2/volumes" Oct 31 03:57:52.050900 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:52.050807 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q"] Oct 31 03:57:52.096751 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:52.096708 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5d30172-qjg9q"] Oct 31 03:57:53.133806 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.133770 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a274751-c231-417f-a8c1-799655f5ef63" path="/var/lib/kubelet/pods/3a274751-c231-417f-a8c1-799655f5ef63/volumes" Oct 31 03:57:53.753373 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.753338 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph"] Oct 31 03:57:53.754281 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.754261 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f845d43a-7e97-4986-acf6-cc3524d1b99f" containerName="revoke-keys" Oct 31 03:57:53.754352 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.754283 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f845d43a-7e97-4986-acf6-cc3524d1b99f" containerName="revoke-keys" Oct 31 03:57:53.754445 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.754435 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f845d43a-7e97-4986-acf6-cc3524d1b99f" containerName="revoke-keys" Oct 31 03:57:53.758317 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.758297 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:53.821534 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.821499 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph"] Oct 31 03:57:53.858875 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.858840 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:57:53.858875 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.858847 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:57:53.859166 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.858933 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:57:53.859166 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.859004 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:57:53.953970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.953930 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwj2h\" (UniqueName: \"kubernetes.io/projected/68ff520e-3694-4628-8840-f54fbc9ef3cd-kube-api-access-jwj2h\") pod \"maas-api-revoke-sub-ea6f4e33-xv6ph\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:53.954159 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.954135 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea6f4e33-xv6ph\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:53.954221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:53.954170 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea6f4e33-xv6ph\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:54.056099 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:54.055985 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwj2h\" (UniqueName: \"kubernetes.io/projected/68ff520e-3694-4628-8840-f54fbc9ef3cd-kube-api-access-jwj2h\") pod \"maas-api-revoke-sub-ea6f4e33-xv6ph\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:54.056329 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:54.056149 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea6f4e33-xv6ph\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:54.056329 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:54.056181 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea6f4e33-xv6ph\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:54.056857 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:54.056837 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea6f4e33-xv6ph\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:54.058862 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:54.058836 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea6f4e33-xv6ph\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:54.114197 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:54.114165 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwj2h\" (UniqueName: \"kubernetes.io/projected/68ff520e-3694-4628-8840-f54fbc9ef3cd-kube-api-access-jwj2h\") pod \"maas-api-revoke-sub-ea6f4e33-xv6ph\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:54.373487 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:54.373401 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:54.764778 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:54.764745 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph"] Oct 31 03:57:54.766632 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:57:54.766599 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ff520e_3694_4628_8840_f54fbc9ef3cd.slice/crio-973c30645f6826580f2b4231a1365089295acabd5ceb377c810a0b65f891b513 WatchSource:0}: Error finding container 973c30645f6826580f2b4231a1365089295acabd5ceb377c810a0b65f891b513: Status 404 returned error can't find the container with id 973c30645f6826580f2b4231a1365089295acabd5ceb377c810a0b65f891b513 Oct 31 03:57:55.229243 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.229200 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k"] Oct 31 03:57:55.234207 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.234186 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.275833 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.275785 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.276005 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.275934 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a0c679-9315-4024-a425-ae42d2716f0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.276180 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.276118 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.276305 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.276285 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.276347 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.276323 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.276398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.276366 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a0c679-9315-4024-a425-ae42d2716f0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.276501 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.276478 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.276684 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.276663 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.276765 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.276750 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq54p\" (UniqueName: \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-kube-api-access-bq54p\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.334221 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.334184 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k"] Oct 31 03:57:55.384968 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.384880 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.384968 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.384936 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.385666 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.384984 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a0c679-9315-4024-a425-ae42d2716f0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.385666 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.385024 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.385666 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.385071 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.385666 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.385171 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq54p\" (UniqueName: \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-kube-api-access-bq54p\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.385666 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.385286 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.385666 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.385372 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a0c679-9315-4024-a425-ae42d2716f0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.385666 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.385457 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.385666 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.385564 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.386749 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.386724 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.389779 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.389365 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.389895 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.389778 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.390169 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.390145 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a0c679-9315-4024-a425-ae42d2716f0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.390430 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.390398 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a0c679-9315-4024-a425-ae42d2716f0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.394396 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.394368 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.446559 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.446524 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq54p\" (UniqueName: \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-kube-api-access-bq54p\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.446982 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.446962 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fxl7k\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.469966 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.469931 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" event={"ID":"68ff520e-3694-4628-8840-f54fbc9ef3cd","Type":"ContainerStarted","Data":"10c57f0bb5db4f5e1820a92b55adca54efdd3161d371a8f8db1d86a914f0ae95"} Oct 31 03:57:55.470168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.469985 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" event={"ID":"68ff520e-3694-4628-8840-f54fbc9ef3cd","Type":"ContainerStarted","Data":"973c30645f6826580f2b4231a1365089295acabd5ceb377c810a0b65f891b513"} Oct 31 03:57:55.550273 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.550240 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:55.590277 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:55.590217 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" podStartSLOduration=2.590200888 podStartE2EDuration="2.590200888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:57:55.582788845 +0000 UTC m=+1772.583414068" watchObservedRunningTime="2026-10-31 03:57:55.590200888 +0000 UTC m=+1772.590826140" Oct 31 03:57:56.015987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:56.015960 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k"] Oct 31 03:57:56.017712 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:57:56.017687 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a0c679_9315_4024_a425_ae42d2716f0c.slice/crio-7c6fe55f91b3ae65a45fda69184bbafd88fbf051ac08dd5820e0567bbaab8328 WatchSource:0}: Error finding container 7c6fe55f91b3ae65a45fda69184bbafd88fbf051ac08dd5820e0567bbaab8328: Status 404 returned error can't find the container with id 7c6fe55f91b3ae65a45fda69184bbafd88fbf051ac08dd5820e0567bbaab8328 Oct 31 03:57:56.020495 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:56.020461 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:57:56.020580 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:56.020529 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:57:56.020580 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:56.020560 4426 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:57:56.482762 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:56.482724 4426 generic.go:356] "Generic (PLEG): container finished" podID="68ff520e-3694-4628-8840-f54fbc9ef3cd" containerID="10c57f0bb5db4f5e1820a92b55adca54efdd3161d371a8f8db1d86a914f0ae95" exitCode=0 Oct 31 03:57:56.483227 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:56.482805 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" event={"ID":"68ff520e-3694-4628-8840-f54fbc9ef3cd","Type":"ContainerDied","Data":"10c57f0bb5db4f5e1820a92b55adca54efdd3161d371a8f8db1d86a914f0ae95"} Oct 31 03:57:56.485362 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:56.485334 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" event={"ID":"76a0c679-9315-4024-a425-ae42d2716f0c","Type":"ContainerStarted","Data":"f575c82e8a43286fa4c28acdadfa97e65467a001c7c09631a1cf6941ff73af04"} Oct 31 03:57:56.485533 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:56.485390 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" event={"ID":"76a0c679-9315-4024-a425-ae42d2716f0c","Type":"ContainerStarted","Data":"7c6fe55f91b3ae65a45fda69184bbafd88fbf051ac08dd5820e0567bbaab8328"} Oct 31 03:57:56.550682 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:56.550647 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:57:56.731150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:56.719272 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" podStartSLOduration=1.719249665 podStartE2EDuration="1.719249665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:57:56.713068746 +0000 UTC m=+1773.713693975" watchObservedRunningTime="2026-10-31 03:57:56.719249665 +0000 UTC m=+1773.719874893" Oct 31 03:57:57.551919 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.551880 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fxl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 31 03:57:57.552375 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.551939 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 31 03:57:57.705566 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.705538 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:57.745903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.745864 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-xks-ca\") pod \"68ff520e-3694-4628-8840-f54fbc9ef3cd\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " Oct 31 03:57:57.745903 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.745906 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-openshift-ca\") pod \"68ff520e-3694-4628-8840-f54fbc9ef3cd\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " Oct 31 03:57:57.746235 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.746185 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ff520e-3694-4628-8840-f54fbc9ef3cd-kube-api-access-jwj2h\" (UniqueName: \"kubernetes.io/projected/68ff520e-3694-4628-8840-f54fbc9ef3cd-kube-api-access-jwj2h\") pod \"68ff520e-3694-4628-8840-f54fbc9ef3cd\" (UID: \"68ff520e-3694-4628-8840-f54fbc9ef3cd\") " Oct 31 03:57:57.746425 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.746403 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-openshift-ca" pod "68ff520e-3694-4628-8840-f54fbc9ef3cd" (UID: "68ff520e-3694-4628-8840-f54fbc9ef3cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:57.746821 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.746799 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:57.748019 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.747988 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-xks-ca" pod "68ff520e-3694-4628-8840-f54fbc9ef3cd" (UID: "68ff520e-3694-4628-8840-f54fbc9ef3cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:57.748331 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.748310 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ff520e-3694-4628-8840-f54fbc9ef3cd-kube-api-access-jwj2h" pod "68ff520e-3694-4628-8840-f54fbc9ef3cd" (UID: "68ff520e-3694-4628-8840-f54fbc9ef3cd"). InnerVolumeSpecName "kube-api-access-jwj2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:57.848398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.848360 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwj2h\" (UniqueName: \"kubernetes.io/projected/68ff520e-3694-4628-8840-f54fbc9ef3cd-kube-api-access-jwj2h\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:57.848398 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:57.848400 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ff520e-3694-4628-8840-f54fbc9ef3cd-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:58.506974 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:58.506906 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" event={"ID":"68ff520e-3694-4628-8840-f54fbc9ef3cd","Type":"ContainerDied","Data":"973c30645f6826580f2b4231a1365089295acabd5ceb377c810a0b65f891b513"} Oct 31 03:57:58.506974 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:58.506957 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph" Oct 31 03:57:58.507239 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:58.507006 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="973c30645f6826580f2b4231a1365089295acabd5ceb377c810a0b65f891b513" Oct 31 03:57:58.551763 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:58.551725 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fxl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 31 03:57:58.551942 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:58.551795 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 31 03:57:59.551116 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:59.551080 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fxl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 31 03:57:59.551324 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:57:59.551157 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 31 03:58:00.550577 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:00.550545 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fxl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 31 03:58:00.551063 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:00.550818 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 31 03:58:01.551002 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:01.550964 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fxl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 31 03:58:01.551482 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:01.551172 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 31 03:58:02.551674 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:02.551623 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fxl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 31 03:58:02.552108 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:02.551696 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 31 03:58:02.853933 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:58:02.853831 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:58:03.055293 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:03.055249 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz"] Oct 31 03:58:03.097557 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:03.097519 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e98a52-wc2zz"] Oct 31 03:58:03.135208 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:03.135100 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97cf9175-4a05-412c-b2f5-324e2c227639" path="/var/lib/kubelet/pods/97cf9175-4a05-412c-b2f5-324e2c227639/volumes" Oct 31 03:58:03.551262 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:03.551217 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fxl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 31 03:58:03.551481 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:03.551298 4426 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 31 03:58:04.735762 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:04.734851 4426 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:58:04.735762 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:04.735392 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:58:04.736448 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:04.736428 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 03:58:10.048778 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:10.048735 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5"] Oct 31 03:58:10.094857 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:10.094818 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34587b81-6nhv5"] Oct 31 03:58:11.135584 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:11.135547 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87beeef2-c988-4981-8f94-955df7f7879b" path="/var/lib/kubelet/pods/87beeef2-c988-4981-8f94-955df7f7879b/volumes" Oct 31 03:58:13.613789 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:13.613693 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:58:16.060149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:16.059824 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk"] Oct 31 03:58:16.128109 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:16.127978 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec6a414-q5fdk"] Oct 31 03:58:17.177150 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:17.176992 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4308b4d0-e562-48c4-9e6b-d95a23e32d27" path="/var/lib/kubelet/pods/4308b4d0-e562-48c4-9e6b-d95a23e32d27/volumes" Oct 31 03:58:21.052512 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:21.052475 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48"] Oct 31 03:58:21.095718 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:21.095682 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539e5bc7-5zs48"] Oct 31 03:58:21.135720 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:21.135687 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c3f2cef-0a5a-489b-9d4a-29c941286885" path="/var/lib/kubelet/pods/0c3f2cef-0a5a-489b-9d4a-29c941286885/volumes" Oct 31 03:58:22.048000 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:22.047966 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8"] Oct 31 03:58:22.093834 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:22.093798 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9e845eb-x2gt8"] Oct 31 03:58:23.132925 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:23.132892 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df025f27-a4ea-4c5b-a078-1773613c06f3" path="/var/lib/kubelet/pods/df025f27-a4ea-4c5b-a078-1773613c06f3/volumes" Oct 31 03:58:23.437148 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:23.437086 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:58:23.442246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:23.442218 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 03:58:24.177536 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:24.177500 4426 scope.go:122] "RemoveContainer" containerID="ff048e83894f1aef6d13961d98ef6ef474b17bac893cfc7671fea18506e9a866" Oct 31 03:58:24.192890 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:24.192852 4426 scope.go:122] "RemoveContainer" containerID="7b01df0b2090391a3e49b9d07f6d836d740bcaca490892a0b32c71f5cef19b3f" Oct 31 03:58:27.049930 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:27.049887 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc"] Oct 31 03:58:27.095260 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:27.095222 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c55df56-zqfbc"] Oct 31 03:58:27.135153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:27.135102 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0cb792-c5b3-430f-bd14-9572930df880" path="/var/lib/kubelet/pods/7e0cb792-c5b3-430f-bd14-9572930df880/volumes" Oct 31 03:58:32.848811 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:58:32.848771 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:58:51.053170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:51.053114 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt"] Oct 31 03:58:51.095781 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:51.095740 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e98011-pjvdt"] Oct 31 03:58:51.151083 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:51.151045 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a35eecf8-6ad3-42d3-9144-85ce3df82f32" path="/var/lib/kubelet/pods/a35eecf8-6ad3-42d3-9144-85ce3df82f32/volumes" Oct 31 03:58:52.078337 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:52.078300 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6"] Oct 31 03:58:52.102108 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:52.102073 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83fe6a26-rbsk6"] Oct 31 03:58:53.136598 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:53.136557 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2503d2fb-448a-48e3-96cf-c37f438101ea" path="/var/lib/kubelet/pods/2503d2fb-448a-48e3-96cf-c37f438101ea/volumes" Oct 31 03:58:56.049904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:56.049863 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph"] Oct 31 03:58:56.057938 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:56.057899 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba7eb0a-x8bph"] Oct 31 03:58:57.137647 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:57.137607 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9225d75c-1d9e-45e0-a415-34d8b60be54b" path="/var/lib/kubelet/pods/9225d75c-1d9e-45e0-a415-34d8b60be54b/volumes" Oct 31 03:58:58.060404 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:58.060359 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw"] Oct 31 03:58:58.107484 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:58.107452 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbdc3ef4-mfmrw"] Oct 31 03:58:59.151151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:58:59.150470 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98d149f-1eec-4e20-bcd7-2a4d7a89bb12" path="/var/lib/kubelet/pods/c98d149f-1eec-4e20-bcd7-2a4d7a89bb12/volumes" Oct 31 03:59:00.382907 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.382861 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj"] Oct 31 03:59:00.384565 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.384538 4426 cpu_manager.go:414] "containerMap: removing container" podUID="68ff520e-3694-4628-8840-f54fbc9ef3cd" containerName="revoke-keys" Oct 31 03:59:00.384756 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.384741 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ff520e-3694-4628-8840-f54fbc9ef3cd" containerName="revoke-keys" Oct 31 03:59:00.385360 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.385341 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ff520e-3694-4628-8840-f54fbc9ef3cd" containerName="revoke-keys" Oct 31 03:59:00.393306 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.393276 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 03:59:00.448973 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.448939 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-22e20532\"" Oct 31 03:59:00.462362 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.452876 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06bb90ba-05f7-44c4-a917-35de0ac617dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj\" (UID: \"06bb90ba-05f7-44c4-a917-35de0ac617dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 03:59:00.473867 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.467552 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t229q\" (UniqueName: \"kubernetes.io/projected/06bb90ba-05f7-44c4-a917-35de0ac617dd-kube-api-access-t229q\") pod \"payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj\" (UID: \"06bb90ba-05f7-44c4-a917-35de0ac617dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 03:59:00.494830 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.494639 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj"] Oct 31 03:59:00.507538 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.507501 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-22e20532-dockercfg-q7vj6\"" Oct 31 03:59:00.568014 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.567467 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d"] Oct 31 03:59:00.577599 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.577568 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 03:59:00.592985 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.592947 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t229q\" (UniqueName: \"kubernetes.io/projected/06bb90ba-05f7-44c4-a917-35de0ac617dd-kube-api-access-t229q\") pod \"payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj\" (UID: \"06bb90ba-05f7-44c4-a917-35de0ac617dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 03:59:00.593273 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.593251 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06bb90ba-05f7-44c4-a917-35de0ac617dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj\" (UID: \"06bb90ba-05f7-44c4-a917-35de0ac617dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 03:59:00.594448 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.594279 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06bb90ba-05f7-44c4-a917-35de0ac617dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj\" (UID: \"06bb90ba-05f7-44c4-a917-35de0ac617dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 03:59:00.598543 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.598519 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d"] Oct 31 03:59:00.687155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.677908 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t229q\" (UniqueName: \"kubernetes.io/projected/06bb90ba-05f7-44c4-a917-35de0ac617dd-kube-api-access-t229q\") pod \"payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj\" (UID: \"06bb90ba-05f7-44c4-a917-35de0ac617dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 03:59:00.698759 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.696901 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zckbd\" (UniqueName: \"kubernetes.io/projected/514aed0d-15b6-4569-9139-2362d7f0ca38-kube-api-access-zckbd\") pod \"payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d\" (UID: \"514aed0d-15b6-4569-9139-2362d7f0ca38\") " pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 03:59:00.698759 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.696962 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514aed0d-15b6-4569-9139-2362d7f0ca38-plugins-config-volume\") pod \"payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d\" (UID: \"514aed0d-15b6-4569-9139-2362d7f0ca38\") " pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 03:59:00.799743 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.799697 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zckbd\" (UniqueName: \"kubernetes.io/projected/514aed0d-15b6-4569-9139-2362d7f0ca38-kube-api-access-zckbd\") pod \"payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d\" (UID: \"514aed0d-15b6-4569-9139-2362d7f0ca38\") " pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 03:59:00.799958 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.799760 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514aed0d-15b6-4569-9139-2362d7f0ca38-plugins-config-volume\") pod \"payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d\" (UID: \"514aed0d-15b6-4569-9139-2362d7f0ca38\") " pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 03:59:00.801107 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.801078 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514aed0d-15b6-4569-9139-2362d7f0ca38-plugins-config-volume\") pod \"payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d\" (UID: \"514aed0d-15b6-4569-9139-2362d7f0ca38\") " pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 03:59:00.821490 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.821446 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 03:59:00.857420 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.857389 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zckbd\" (UniqueName: \"kubernetes.io/projected/514aed0d-15b6-4569-9139-2362d7f0ca38-kube-api-access-zckbd\") pod \"payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d\" (UID: \"514aed0d-15b6-4569-9139-2362d7f0ca38\") " pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 03:59:00.903170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:00.898248 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 03:59:01.067555 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:01.067484 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7"] Oct 31 03:59:01.080137 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:01.079507 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f73a7b-zwtg7"] Oct 31 03:59:01.153744 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:01.152775 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a75e294-8bd3-4629-9ce9-45d4203215a2" path="/var/lib/kubelet/pods/0a75e294-8bd3-4629-9ce9-45d4203215a2/volumes" Oct 31 03:59:01.237873 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:01.237847 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj"] Oct 31 03:59:01.241172 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:59:01.241136 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bb90ba_05f7_44c4_a917_35de0ac617dd.slice/crio-712b0405eb50addc184c889ee8891cf0934a5a992fbe218c22a1a947b125bdcb WatchSource:0}: Error finding container 712b0405eb50addc184c889ee8891cf0934a5a992fbe218c22a1a947b125bdcb: Status 404 returned error can't find the container with id 712b0405eb50addc184c889ee8891cf0934a5a992fbe218c22a1a947b125bdcb Oct 31 03:59:01.429447 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:01.429409 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d"] Oct 31 03:59:01.436045 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:59:01.436016 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514aed0d_15b6_4569_9139_2362d7f0ca38.slice/crio-fb8762786b10508b889f3a636bac53ac06aec5332444389b8e9e28b7bd162569 WatchSource:0}: Error finding container fb8762786b10508b889f3a636bac53ac06aec5332444389b8e9e28b7bd162569: Status 404 returned error can't find the container with id fb8762786b10508b889f3a636bac53ac06aec5332444389b8e9e28b7bd162569 Oct 31 03:59:01.463951 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:01.463913 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" event={"ID":"06bb90ba-05f7-44c4-a917-35de0ac617dd","Type":"ContainerStarted","Data":"3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598"} Oct 31 03:59:01.464106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:01.463981 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" event={"ID":"06bb90ba-05f7-44c4-a917-35de0ac617dd","Type":"ContainerStarted","Data":"712b0405eb50addc184c889ee8891cf0934a5a992fbe218c22a1a947b125bdcb"} Oct 31 03:59:01.464990 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:01.464967 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 03:59:01.466932 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:01.466902 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" event={"ID":"514aed0d-15b6-4569-9139-2362d7f0ca38","Type":"ContainerStarted","Data":"fb8762786b10508b889f3a636bac53ac06aec5332444389b8e9e28b7bd162569"} Oct 31 03:59:01.577802 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:01.577742 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" podStartSLOduration=1.5777240209999999 podStartE2EDuration="1.577724021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:01.573440113 +0000 UTC m=+1838.574065341" watchObservedRunningTime="2026-10-31 03:59:01.577724021 +0000 UTC m=+1838.578349295" Oct 31 03:59:02.479444 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:02.479400 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" event={"ID":"514aed0d-15b6-4569-9139-2362d7f0ca38","Type":"ContainerStarted","Data":"83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741"} Oct 31 03:59:02.479970 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:02.479514 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 03:59:02.860052 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:59:02.859948 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:59:05.492958 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.492886 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" podStartSLOduration=5.492864953 podStartE2EDuration="5.492864953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:02.587961176 +0000 UTC m=+1839.588586401" watchObservedRunningTime="2026-10-31 03:59:05.492864953 +0000 UTC m=+1842.493490181" Oct 31 03:59:05.500701 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.500668 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4"] Oct 31 03:59:05.512825 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.510363 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 03:59:05.562737 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.562289 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4"] Oct 31 03:59:05.604003 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.601328 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-b4a3a5\"" Oct 31 03:59:05.604003 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.601476 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-b4a3a5-dockercfg-nrp47\"" Oct 31 03:59:05.616330 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.616063 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mznf\" (UniqueName: \"kubernetes.io/projected/f2ea1ec7-6b32-43ab-9738-3deaf786a844-kube-api-access-6mznf\") pod \"payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4\" (UID: \"f2ea1ec7-6b32-43ab-9738-3deaf786a844\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 03:59:05.616330 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.616291 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ea1ec7-6b32-43ab-9738-3deaf786a844-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4\" (UID: \"f2ea1ec7-6b32-43ab-9738-3deaf786a844\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 03:59:05.723373 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.723282 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mznf\" (UniqueName: \"kubernetes.io/projected/f2ea1ec7-6b32-43ab-9738-3deaf786a844-kube-api-access-6mznf\") pod \"payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4\" (UID: \"f2ea1ec7-6b32-43ab-9738-3deaf786a844\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 03:59:05.723545 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.723450 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ea1ec7-6b32-43ab-9738-3deaf786a844-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4\" (UID: \"f2ea1ec7-6b32-43ab-9738-3deaf786a844\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 03:59:05.724893 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.724867 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ea1ec7-6b32-43ab-9738-3deaf786a844-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4\" (UID: \"f2ea1ec7-6b32-43ab-9738-3deaf786a844\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 03:59:05.751168 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.743207 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62"] Oct 31 03:59:05.777401 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.775569 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62"] Oct 31 03:59:05.777401 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.775695 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 03:59:05.812468 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.812431 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mznf\" (UniqueName: \"kubernetes.io/projected/f2ea1ec7-6b32-43ab-9738-3deaf786a844-kube-api-access-6mznf\") pod \"payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4\" (UID: \"f2ea1ec7-6b32-43ab-9738-3deaf786a844\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 03:59:05.825492 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.825452 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz64f\" (UniqueName: \"kubernetes.io/projected/e2851d33-a4f5-4f39-b19a-990a2dd277dd-kube-api-access-jz64f\") pod \"payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62\" (UID: \"e2851d33-a4f5-4f39-b19a-990a2dd277dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 03:59:05.825779 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.825754 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2851d33-a4f5-4f39-b19a-990a2dd277dd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62\" (UID: \"e2851d33-a4f5-4f39-b19a-990a2dd277dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 03:59:05.853270 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.853230 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 03:59:05.941910 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.928594 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2851d33-a4f5-4f39-b19a-990a2dd277dd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62\" (UID: \"e2851d33-a4f5-4f39-b19a-990a2dd277dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 03:59:05.941910 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.928815 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz64f\" (UniqueName: \"kubernetes.io/projected/e2851d33-a4f5-4f39-b19a-990a2dd277dd-kube-api-access-jz64f\") pod \"payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62\" (UID: \"e2851d33-a4f5-4f39-b19a-990a2dd277dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 03:59:05.941910 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.932479 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2851d33-a4f5-4f39-b19a-990a2dd277dd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62\" (UID: \"e2851d33-a4f5-4f39-b19a-990a2dd277dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 03:59:05.991079 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:05.991046 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz64f\" (UniqueName: \"kubernetes.io/projected/e2851d33-a4f5-4f39-b19a-990a2dd277dd-kube-api-access-jz64f\") pod \"payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62\" (UID: \"e2851d33-a4f5-4f39-b19a-990a2dd277dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 03:59:06.060183 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:06.058814 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8"] Oct 31 03:59:06.067241 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:06.067205 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d9d4f8-72kr8"] Oct 31 03:59:06.138364 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:06.138094 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 03:59:06.415275 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:06.415236 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4"] Oct 31 03:59:06.425985 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:59:06.425950 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ea1ec7_6b32_43ab_9738_3deaf786a844.slice/crio-daac39d29cf0253cb49ab03e9e8174aeb5e4a465277f9a1ad45b26a70de5f06f WatchSource:0}: Error finding container daac39d29cf0253cb49ab03e9e8174aeb5e4a465277f9a1ad45b26a70de5f06f: Status 404 returned error can't find the container with id daac39d29cf0253cb49ab03e9e8174aeb5e4a465277f9a1ad45b26a70de5f06f Oct 31 03:59:06.541170 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:06.541098 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" event={"ID":"f2ea1ec7-6b32-43ab-9738-3deaf786a844","Type":"ContainerStarted","Data":"daac39d29cf0253cb49ab03e9e8174aeb5e4a465277f9a1ad45b26a70de5f06f"} Oct 31 03:59:06.757010 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:06.756979 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62"] Oct 31 03:59:06.760334 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:59:06.760300 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2851d33_a4f5_4f39_b19a_990a2dd277dd.slice/crio-daf7bc0cc6e29b463e34e8cb55ebe07984f5b226bfd46afddd068923a11f27f4 WatchSource:0}: Error finding container daf7bc0cc6e29b463e34e8cb55ebe07984f5b226bfd46afddd068923a11f27f4: Status 404 returned error can't find the container with id daf7bc0cc6e29b463e34e8cb55ebe07984f5b226bfd46afddd068923a11f27f4 Oct 31 03:59:07.140978 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:07.140931 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2370901-ce84-4e56-a7aa-213c99b20923" path="/var/lib/kubelet/pods/f2370901-ce84-4e56-a7aa-213c99b20923/volumes" Oct 31 03:59:07.553969 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:07.553874 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" event={"ID":"f2ea1ec7-6b32-43ab-9738-3deaf786a844","Type":"ContainerStarted","Data":"93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea"} Oct 31 03:59:07.554622 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:07.554583 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 03:59:07.556344 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:07.556240 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" event={"ID":"e2851d33-a4f5-4f39-b19a-990a2dd277dd","Type":"ContainerStarted","Data":"3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8"} Oct 31 03:59:07.556344 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:07.556313 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" event={"ID":"e2851d33-a4f5-4f39-b19a-990a2dd277dd","Type":"ContainerStarted","Data":"daf7bc0cc6e29b463e34e8cb55ebe07984f5b226bfd46afddd068923a11f27f4"} Oct 31 03:59:07.556503 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:07.556357 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 03:59:07.664766 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:07.664700 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" podStartSLOduration=2.664683278 podStartE2EDuration="2.664683278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:07.661967518 +0000 UTC m=+1844.662592739" watchObservedRunningTime="2026-10-31 03:59:07.664683278 +0000 UTC m=+1844.665308507" Oct 31 03:59:07.779941 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:07.779866 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" podStartSLOduration=2.779845582 podStartE2EDuration="2.779845582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:07.771235393 +0000 UTC m=+1844.771860621" watchObservedRunningTime="2026-10-31 03:59:07.779845582 +0000 UTC m=+1844.780470812" Oct 31 03:59:10.050471 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:10.050434 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh"] Oct 31 03:59:10.095478 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:10.095443 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a12f38-f6bsh"] Oct 31 03:59:11.136063 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:11.136028 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27560d7d-ccf1-4bf1-a051-bfa1f0666e3a" path="/var/lib/kubelet/pods/27560d7d-ccf1-4bf1-a051-bfa1f0666e3a/volumes" Oct 31 03:59:12.383873 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.383822 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw"] Oct 31 03:59:12.391504 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.391476 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 03:59:12.427948 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.427907 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz"] Oct 31 03:59:12.438955 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.438924 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:12.473375 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.473336 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw"] Oct 31 03:59:12.501162 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.500665 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f8a527-8cc0-44f4-a09a-3703e4444154-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw\" (UID: \"26f8a527-8cc0-44f4-a09a-3703e4444154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 03:59:12.501162 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.500773 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42vv4\" (UniqueName: \"kubernetes.io/projected/26f8a527-8cc0-44f4-a09a-3703e4444154-kube-api-access-42vv4\") pod \"payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw\" (UID: \"26f8a527-8cc0-44f4-a09a-3703e4444154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 03:59:12.561218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.533618 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-976cea-dockercfg-s6s4x\"" Oct 31 03:59:12.561218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.533759 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-976cea\"" Oct 31 03:59:12.562658 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.562625 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:59:12.562939 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.562920 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:12.563209 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.563107 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:59:12.563638 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.563618 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:59:12.615334 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.615289 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvj6\" (UniqueName: \"kubernetes.io/projected/b62c0e50-46ac-4d91-9976-5471764fc6ef-kube-api-access-npvj6\") pod \"maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:12.636208 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.617503 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f8a527-8cc0-44f4-a09a-3703e4444154-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw\" (UID: \"26f8a527-8cc0-44f4-a09a-3703e4444154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 03:59:12.636208 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.617596 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:12.636208 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.617693 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42vv4\" (UniqueName: \"kubernetes.io/projected/26f8a527-8cc0-44f4-a09a-3703e4444154-kube-api-access-42vv4\") pod \"payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw\" (UID: \"26f8a527-8cc0-44f4-a09a-3703e4444154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 03:59:12.636208 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.617891 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:12.636208 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.619233 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f8a527-8cc0-44f4-a09a-3703e4444154-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw\" (UID: \"26f8a527-8cc0-44f4-a09a-3703e4444154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 03:59:12.652488 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.652223 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz"] Oct 31 03:59:12.700217 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.698911 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 03:59:12.721351 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.721305 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvj6\" (UniqueName: \"kubernetes.io/projected/b62c0e50-46ac-4d91-9976-5471764fc6ef-kube-api-access-npvj6\") pod \"maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:12.722370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.721397 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:12.722370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.721569 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:12.726438 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.724118 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:12.726438 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.725597 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42vv4\" (UniqueName: \"kubernetes.io/projected/26f8a527-8cc0-44f4-a09a-3703e4444154-kube-api-access-42vv4\") pod \"payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw\" (UID: \"26f8a527-8cc0-44f4-a09a-3703e4444154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 03:59:12.742085 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.741792 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:12.742627 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.742594 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7"] Oct 31 03:59:12.752430 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.752402 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7"] Oct 31 03:59:12.752597 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.752493 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 03:59:12.781188 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.781150 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvj6\" (UniqueName: \"kubernetes.io/projected/b62c0e50-46ac-4d91-9976-5471764fc6ef-kube-api-access-npvj6\") pod \"maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:12.888748 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.888651 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:12.935306 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.935262 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjv8\" (UniqueName: \"kubernetes.io/projected/283c4e57-10b6-46ad-92b6-d490dfff7db2-kube-api-access-bgjv8\") pod \"payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7\" (UID: \"283c4e57-10b6-46ad-92b6-d490dfff7db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 03:59:12.935514 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:12.935409 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/283c4e57-10b6-46ad-92b6-d490dfff7db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7\" (UID: \"283c4e57-10b6-46ad-92b6-d490dfff7db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 03:59:13.068198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.020424 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 03:59:13.068198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.038253 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjv8\" (UniqueName: \"kubernetes.io/projected/283c4e57-10b6-46ad-92b6-d490dfff7db2-kube-api-access-bgjv8\") pod \"payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7\" (UID: \"283c4e57-10b6-46ad-92b6-d490dfff7db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 03:59:13.068198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.038330 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/283c4e57-10b6-46ad-92b6-d490dfff7db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7\" (UID: \"283c4e57-10b6-46ad-92b6-d490dfff7db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 03:59:13.068198 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.040246 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/283c4e57-10b6-46ad-92b6-d490dfff7db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7\" (UID: \"283c4e57-10b6-46ad-92b6-d490dfff7db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 03:59:13.145643 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.130620 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjv8\" (UniqueName: \"kubernetes.io/projected/283c4e57-10b6-46ad-92b6-d490dfff7db2-kube-api-access-bgjv8\") pod \"payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7\" (UID: \"283c4e57-10b6-46ad-92b6-d490dfff7db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 03:59:13.356692 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.356647 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz"] Oct 31 03:59:13.360244 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:59:13.360209 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62c0e50_46ac_4d91_9976_5471764fc6ef.slice/crio-c3ae2e860a761ff20d433fe4d2dd4d0fe98d0179d89212ca6b54fc7ed2ef8bcb WatchSource:0}: Error finding container c3ae2e860a761ff20d433fe4d2dd4d0fe98d0179d89212ca6b54fc7ed2ef8bcb: Status 404 returned error can't find the container with id c3ae2e860a761ff20d433fe4d2dd4d0fe98d0179d89212ca6b54fc7ed2ef8bcb Oct 31 03:59:13.370326 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.370298 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 03:59:13.493552 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.492848 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 03:59:13.503811 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.503787 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw"] Oct 31 03:59:13.535240 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:59:13.520020 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f8a527_8cc0_44f4_a09a_3703e4444154.slice/crio-43e3ef5c4035553606126e8e91d802fccce390ed895094018cc4c61fd3c1a928 WatchSource:0}: Error finding container 43e3ef5c4035553606126e8e91d802fccce390ed895094018cc4c61fd3c1a928: Status 404 returned error can't find the container with id 43e3ef5c4035553606126e8e91d802fccce390ed895094018cc4c61fd3c1a928 Oct 31 03:59:13.754370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.754274 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" event={"ID":"26f8a527-8cc0-44f4-a09a-3703e4444154","Type":"ContainerStarted","Data":"14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d"} Oct 31 03:59:13.754370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.754346 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 03:59:13.754370 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.754370 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" event={"ID":"26f8a527-8cc0-44f4-a09a-3703e4444154","Type":"ContainerStarted","Data":"43e3ef5c4035553606126e8e91d802fccce390ed895094018cc4c61fd3c1a928"} Oct 31 03:59:13.756045 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.756013 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" event={"ID":"b62c0e50-46ac-4d91-9976-5471764fc6ef","Type":"ContainerStarted","Data":"c3ae2e860a761ff20d433fe4d2dd4d0fe98d0179d89212ca6b54fc7ed2ef8bcb"} Oct 31 03:59:13.775660 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.775628 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7"] Oct 31 03:59:13.817310 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:59:13.817260 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283c4e57_10b6_46ad_92b6_d490dfff7db2.slice/crio-d1022bf2ddc089e68081ff24fe1194e2a0eeb67f82fe052ac261383fca35a9d7 WatchSource:0}: Error finding container d1022bf2ddc089e68081ff24fe1194e2a0eeb67f82fe052ac261383fca35a9d7: Status 404 returned error can't find the container with id d1022bf2ddc089e68081ff24fe1194e2a0eeb67f82fe052ac261383fca35a9d7 Oct 31 03:59:13.868115 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:13.868064 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" podStartSLOduration=1.86804901 podStartE2EDuration="1.86804901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:13.862990038 +0000 UTC m=+1850.863615269" watchObservedRunningTime="2026-10-31 03:59:13.86804901 +0000 UTC m=+1850.868674271" Oct 31 03:59:14.770705 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:14.770584 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" event={"ID":"283c4e57-10b6-46ad-92b6-d490dfff7db2","Type":"ContainerStarted","Data":"af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9"} Oct 31 03:59:14.770705 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:14.770654 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 03:59:14.770705 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:14.770678 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" event={"ID":"283c4e57-10b6-46ad-92b6-d490dfff7db2","Type":"ContainerStarted","Data":"d1022bf2ddc089e68081ff24fe1194e2a0eeb67f82fe052ac261383fca35a9d7"} Oct 31 03:59:14.773008 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:14.772979 4426 generic.go:356] "Generic (PLEG): container finished" podID="b62c0e50-46ac-4d91-9976-5471764fc6ef" containerID="b7614ff91c8144bdfd16078c9107b7a8077c2e565ee15104c5408848bded5193" exitCode=0 Oct 31 03:59:14.773106 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:14.773054 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" event={"ID":"b62c0e50-46ac-4d91-9976-5471764fc6ef","Type":"ContainerDied","Data":"b7614ff91c8144bdfd16078c9107b7a8077c2e565ee15104c5408848bded5193"} Oct 31 03:59:14.881725 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:14.881659 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" podStartSLOduration=2.881639835 podStartE2EDuration="2.881639835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:14.880364581 +0000 UTC m=+1851.880989808" watchObservedRunningTime="2026-10-31 03:59:14.881639835 +0000 UTC m=+1851.882265064" Oct 31 03:59:16.014164 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.010984 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:16.057009 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.053625 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m"] Oct 31 03:59:16.118153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.117749 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-openshift-ca\") pod \"b62c0e50-46ac-4d91-9976-5471764fc6ef\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " Oct 31 03:59:16.118153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.117822 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b62c0e50-46ac-4d91-9976-5471764fc6ef-kube-api-access-npvj6\" (UniqueName: \"kubernetes.io/projected/b62c0e50-46ac-4d91-9976-5471764fc6ef-kube-api-access-npvj6\") pod \"b62c0e50-46ac-4d91-9976-5471764fc6ef\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " Oct 31 03:59:16.118153 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.117947 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-xks-ca\") pod \"b62c0e50-46ac-4d91-9976-5471764fc6ef\" (UID: \"b62c0e50-46ac-4d91-9976-5471764fc6ef\") " Oct 31 03:59:16.123258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.118464 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-openshift-ca" pod "b62c0e50-46ac-4d91-9976-5471764fc6ef" (UID: "b62c0e50-46ac-4d91-9976-5471764fc6ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:16.123258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.118923 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:16.123258 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.123055 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17869026-g745m"] Oct 31 03:59:16.145359 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.144408 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62c0e50-46ac-4d91-9976-5471764fc6ef-kube-api-access-npvj6" pod "b62c0e50-46ac-4d91-9976-5471764fc6ef" (UID: "b62c0e50-46ac-4d91-9976-5471764fc6ef"). InnerVolumeSpecName "kube-api-access-npvj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:16.152301 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.145707 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-xks-ca" pod "b62c0e50-46ac-4d91-9976-5471764fc6ef" (UID: "b62c0e50-46ac-4d91-9976-5471764fc6ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:16.246102 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.243500 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npvj6\" (UniqueName: \"kubernetes.io/projected/b62c0e50-46ac-4d91-9976-5471764fc6ef-kube-api-access-npvj6\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:16.246102 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.243545 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62c0e50-46ac-4d91-9976-5471764fc6ef-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:16.795190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.795161 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" Oct 31 03:59:16.795190 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.795176 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz" event={"ID":"b62c0e50-46ac-4d91-9976-5471764fc6ef","Type":"ContainerDied","Data":"c3ae2e860a761ff20d433fe4d2dd4d0fe98d0179d89212ca6b54fc7ed2ef8bcb"} Oct 31 03:59:16.795406 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:16.795231 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3ae2e860a761ff20d433fe4d2dd4d0fe98d0179d89212ca6b54fc7ed2ef8bcb" Oct 31 03:59:17.132900 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:17.132805 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba5953b-322e-46f3-913e-637454ea624a" path="/var/lib/kubelet/pods/dba5953b-322e-46f3-913e-637454ea624a/volumes" Oct 31 03:59:17.704607 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:17.704573 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 03:59:18.580360 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:18.580327 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 03:59:18.580923 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:18.580427 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 03:59:22.047569 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:22.047531 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx"] Oct 31 03:59:22.053519 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:22.053491 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a85fb4fb-dcmvx"] Oct 31 03:59:23.133543 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:23.133509 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41025c7d-8f03-4f3e-9926-aa3408bf86de" path="/var/lib/kubelet/pods/41025c7d-8f03-4f3e-9926-aa3408bf86de/volumes" Oct 31 03:59:24.237252 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:24.237216 4426 scope.go:122] "RemoveContainer" containerID="31cb679998ebd7bbb6c34c63fb8fb2077afd2ad063db084df2e2984419ed7e4f" Oct 31 03:59:24.254931 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:24.254899 4426 scope.go:122] "RemoveContainer" containerID="0f2f687120e185e2fc30c4a2bb0ce490493f171af078df9a3291a628642dfc88" Oct 31 03:59:24.267202 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:24.267117 4426 scope.go:122] "RemoveContainer" containerID="187dc2dc83e0bd9c790bdd549e434a6f3c6245dbe306ecf2df9bf0178a54e86d" Oct 31 03:59:24.278670 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:24.278644 4426 scope.go:122] "RemoveContainer" containerID="31f8d35b1150d601b844fb73481e4718e6074e109dd3097e21cb1ffdfe3cfa34" Oct 31 03:59:24.290108 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:24.290084 4426 scope.go:122] "RemoveContainer" containerID="c3a4f7b4d7f1468a00fc65705aa84fcef313e4863db2fa0cb116ae240d3fa69f" Oct 31 03:59:24.301834 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:24.301813 4426 scope.go:122] "RemoveContainer" containerID="35f6ec230b75c4eb30edfd0eed85c4ebf235a29c5ae63a44a413c2bfdcbdd2e7" Oct 31 03:59:24.312967 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:24.312947 4426 scope.go:122] "RemoveContainer" containerID="d0bbedf7488a9fcd47a4beffa492ba7293d23e2698786c85d2101d0404062f00" Oct 31 03:59:24.323904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:24.323880 4426 scope.go:122] "RemoveContainer" containerID="829c203cf421750f5f6807061851f56eb31ab36691e5713fa5d56a145b7073e5" Oct 31 03:59:24.335979 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:24.335954 4426 scope.go:122] "RemoveContainer" containerID="c06394432d110b463f5516da73cf228b1631d0da4339da75abf5253a9ce2b4a2" Oct 31 03:59:24.777486 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:24.777454 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 03:59:25.057956 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.057867 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2"] Oct 31 03:59:25.059179 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.059105 4426 cpu_manager.go:414] "containerMap: removing container" podUID="b62c0e50-46ac-4d91-9976-5471764fc6ef" containerName="revoke-keys" Oct 31 03:59:25.059179 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.059150 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b62c0e50-46ac-4d91-9976-5471764fc6ef" containerName="revoke-keys" Oct 31 03:59:25.059775 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.059415 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="b62c0e50-46ac-4d91-9976-5471764fc6ef" containerName="revoke-keys" Oct 31 03:59:25.067963 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.067916 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:25.119832 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.119485 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2"] Oct 31 03:59:25.147246 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.147210 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:59:25.147904 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.147882 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 03:59:25.148031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.147956 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:25.148031 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.148015 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:59:25.219062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.217061 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:25.219062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.217607 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:25.219062 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.217771 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ls8k\" (UniqueName: \"kubernetes.io/projected/2e2ac1b0-6998-4daa-8f90-df002347e7b5-kube-api-access-7ls8k\") pod \"maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:25.321891 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.321798 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:25.321891 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.321873 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ls8k\" (UniqueName: \"kubernetes.io/projected/2e2ac1b0-6998-4daa-8f90-df002347e7b5-kube-api-access-7ls8k\") pod \"maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:25.322354 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.321905 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:25.323114 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.323083 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:25.324863 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.324835 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:25.382803 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.382469 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ls8k\" (UniqueName: \"kubernetes.io/projected/2e2ac1b0-6998-4daa-8f90-df002347e7b5-kube-api-access-7ls8k\") pod \"maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:25.404441 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.387085 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:25.787701 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.787670 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 03:59:25.809974 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.809951 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2"] Oct 31 03:59:25.813301 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:59:25.813269 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2ac1b0_6998_4daa_8f90_df002347e7b5.slice/crio-d33c4bc7ee31ca67a048c0c40df8cae81923d905496b3362f34f537d75807e26 WatchSource:0}: Error finding container d33c4bc7ee31ca67a048c0c40df8cae81923d905496b3362f34f537d75807e26: Status 404 returned error can't find the container with id d33c4bc7ee31ca67a048c0c40df8cae81923d905496b3362f34f537d75807e26 Oct 31 03:59:25.954304 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:25.954260 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" event={"ID":"2e2ac1b0-6998-4daa-8f90-df002347e7b5","Type":"ContainerStarted","Data":"d33c4bc7ee31ca67a048c0c40df8cae81923d905496b3362f34f537d75807e26"} Oct 31 03:59:26.968953 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:26.968904 4426 generic.go:356] "Generic (PLEG): container finished" podID="2e2ac1b0-6998-4daa-8f90-df002347e7b5" containerID="a4de4299a394188b4a6c9bced6fa815ee9910be49fd9b645885204bf42fc1fe3" exitCode=0 Oct 31 03:59:26.969515 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:26.969149 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" event={"ID":"2e2ac1b0-6998-4daa-8f90-df002347e7b5","Type":"ContainerDied","Data":"a4de4299a394188b4a6c9bced6fa815ee9910be49fd9b645885204bf42fc1fe3"} Oct 31 03:59:27.052211 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:27.052078 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l"] Oct 31 03:59:27.086805 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:27.086770 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx"] Oct 31 03:59:27.098828 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:27.098788 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c61a59f-mx2gx"] Oct 31 03:59:27.108642 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:27.108605 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14769dc-mfg5l"] Oct 31 03:59:27.138625 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:27.138476 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1397f512-8b19-4e94-881c-6030d5953ee7" path="/var/lib/kubelet/pods/1397f512-8b19-4e94-881c-6030d5953ee7/volumes" Oct 31 03:59:27.139228 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:27.139188 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc366446-32f5-413f-896f-58b60f082b5e" path="/var/lib/kubelet/pods/cc366446-32f5-413f-896f-58b60f082b5e/volumes" Oct 31 03:59:28.173303 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:28.173271 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:28.281589 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:28.281451 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-openshift-ca\") pod \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " Oct 31 03:59:28.281589 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:28.281542 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2ac1b0-6998-4daa-8f90-df002347e7b5-kube-api-access-7ls8k\" (UniqueName: \"kubernetes.io/projected/2e2ac1b0-6998-4daa-8f90-df002347e7b5-kube-api-access-7ls8k\") pod \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " Oct 31 03:59:28.281823 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:28.281701 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-xks-ca\") pod \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\" (UID: \"2e2ac1b0-6998-4daa-8f90-df002347e7b5\") " Oct 31 03:59:28.281881 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:28.281846 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-openshift-ca" pod "2e2ac1b0-6998-4daa-8f90-df002347e7b5" (UID: "2e2ac1b0-6998-4daa-8f90-df002347e7b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:28.283669 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:28.283634 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:28.284777 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:28.284751 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-xks-ca" pod "2e2ac1b0-6998-4daa-8f90-df002347e7b5" (UID: "2e2ac1b0-6998-4daa-8f90-df002347e7b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:28.287348 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:28.287324 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2ac1b0-6998-4daa-8f90-df002347e7b5-kube-api-access-7ls8k" pod "2e2ac1b0-6998-4daa-8f90-df002347e7b5" (UID: "2e2ac1b0-6998-4daa-8f90-df002347e7b5"). InnerVolumeSpecName "kube-api-access-7ls8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:28.385909 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:28.385866 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ls8k\" (UniqueName: \"kubernetes.io/projected/2e2ac1b0-6998-4daa-8f90-df002347e7b5-kube-api-access-7ls8k\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:28.385909 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:28.385903 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2ac1b0-6998-4daa-8f90-df002347e7b5-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:29.035265 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.035218 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" event={"ID":"2e2ac1b0-6998-4daa-8f90-df002347e7b5","Type":"ContainerDied","Data":"d33c4bc7ee31ca67a048c0c40df8cae81923d905496b3362f34f537d75807e26"} Oct 31 03:59:29.035844 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.035283 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d33c4bc7ee31ca67a048c0c40df8cae81923d905496b3362f34f537d75807e26" Oct 31 03:59:29.035844 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.035234 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2" Oct 31 03:59:29.074863 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.074833 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm"] Oct 31 03:59:29.075621 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.075605 4426 cpu_manager.go:414] "containerMap: removing container" podUID="2e2ac1b0-6998-4daa-8f90-df002347e7b5" containerName="revoke-keys" Oct 31 03:59:29.075677 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.075622 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2ac1b0-6998-4daa-8f90-df002347e7b5" containerName="revoke-keys" Oct 31 03:59:29.076885 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.076855 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2ac1b0-6998-4daa-8f90-df002347e7b5" containerName="revoke-keys" Oct 31 03:59:29.085891 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.085854 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 03:59:29.145152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.135423 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-440092\"" Oct 31 03:59:29.145152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.136903 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994007af-8bf3-414d-9492-7934ccdbcd22-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm\" (UID: \"994007af-8bf3-414d-9492-7934ccdbcd22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 03:59:29.145152 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.136946 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjbn6\" (UniqueName: \"kubernetes.io/projected/994007af-8bf3-414d-9492-7934ccdbcd22-kube-api-access-gjbn6\") pod \"payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm\" (UID: \"994007af-8bf3-414d-9492-7934ccdbcd22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 03:59:29.161177 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.155386 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-440092-dockercfg-nsclq\"" Oct 31 03:59:29.161177 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.158814 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm"] Oct 31 03:59:29.192144 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.190562 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js"] Oct 31 03:59:29.206054 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.204706 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 03:59:29.239218 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.239114 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb84fa0-c690-40cd-b19e-db6b1767bc34-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-440092-5d59845559-827js\" (UID: \"dfb84fa0-c690-40cd-b19e-db6b1767bc34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 03:59:29.239469 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.239253 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994007af-8bf3-414d-9492-7934ccdbcd22-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm\" (UID: \"994007af-8bf3-414d-9492-7934ccdbcd22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 03:59:29.239469 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.239310 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjbn6\" (UniqueName: \"kubernetes.io/projected/994007af-8bf3-414d-9492-7934ccdbcd22-kube-api-access-gjbn6\") pod \"payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm\" (UID: \"994007af-8bf3-414d-9492-7934ccdbcd22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 03:59:29.239595 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.239475 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5nw\" (UniqueName: \"kubernetes.io/projected/dfb84fa0-c690-40cd-b19e-db6b1767bc34-kube-api-access-bx5nw\") pod \"payload-processing-e2e-shared-b-w5-440092-5d59845559-827js\" (UID: \"dfb84fa0-c690-40cd-b19e-db6b1767bc34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 03:59:29.240781 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.240739 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994007af-8bf3-414d-9492-7934ccdbcd22-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm\" (UID: \"994007af-8bf3-414d-9492-7934ccdbcd22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 03:59:29.256951 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.253966 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js"] Oct 31 03:59:29.351205 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.346922 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjbn6\" (UniqueName: \"kubernetes.io/projected/994007af-8bf3-414d-9492-7934ccdbcd22-kube-api-access-gjbn6\") pod \"payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm\" (UID: \"994007af-8bf3-414d-9492-7934ccdbcd22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 03:59:29.351205 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.348069 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb84fa0-c690-40cd-b19e-db6b1767bc34-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-440092-5d59845559-827js\" (UID: \"dfb84fa0-c690-40cd-b19e-db6b1767bc34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 03:59:29.351205 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.348305 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5nw\" (UniqueName: \"kubernetes.io/projected/dfb84fa0-c690-40cd-b19e-db6b1767bc34-kube-api-access-bx5nw\") pod \"payload-processing-e2e-shared-b-w5-440092-5d59845559-827js\" (UID: \"dfb84fa0-c690-40cd-b19e-db6b1767bc34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 03:59:29.351205 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.349734 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb84fa0-c690-40cd-b19e-db6b1767bc34-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-440092-5d59845559-827js\" (UID: \"dfb84fa0-c690-40cd-b19e-db6b1767bc34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 03:59:29.412802 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.412393 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 03:59:29.476155 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.467750 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5nw\" (UniqueName: \"kubernetes.io/projected/dfb84fa0-c690-40cd-b19e-db6b1767bc34-kube-api-access-bx5nw\") pod \"payload-processing-e2e-shared-b-w5-440092-5d59845559-827js\" (UID: \"dfb84fa0-c690-40cd-b19e-db6b1767bc34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 03:59:29.528188 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.525373 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 03:59:29.927615 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:29.927584 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm"] Oct 31 03:59:29.931878 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:59:29.931847 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994007af_8bf3_414d_9492_7934ccdbcd22.slice/crio-3509b076b31c2e7a041c5a5a8b7331cc249ab1178622736196a521faac186172 WatchSource:0}: Error finding container 3509b076b31c2e7a041c5a5a8b7331cc249ab1178622736196a521faac186172: Status 404 returned error can't find the container with id 3509b076b31c2e7a041c5a5a8b7331cc249ab1178622736196a521faac186172 Oct 31 03:59:30.031204 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:30.031174 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js"] Oct 31 03:59:30.033523 ip-10-0-134-106 kubenswrapper[4426]: W1031 03:59:30.033490 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb84fa0_c690_40cd_b19e_db6b1767bc34.slice/crio-92d4671acbcf81cfcb6c1806a0badb660251a81bccd043314e4d36fd39cbc271 WatchSource:0}: Error finding container 92d4671acbcf81cfcb6c1806a0badb660251a81bccd043314e4d36fd39cbc271: Status 404 returned error can't find the container with id 92d4671acbcf81cfcb6c1806a0badb660251a81bccd043314e4d36fd39cbc271 Oct 31 03:59:30.050098 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:30.050062 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" event={"ID":"994007af-8bf3-414d-9492-7934ccdbcd22","Type":"ContainerStarted","Data":"2a1c5d981309d74e16db3f2b0b6fce2e1758facc4021a5a5598d44736f2a6141"} Oct 31 03:59:30.050380 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:30.050168 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" event={"ID":"994007af-8bf3-414d-9492-7934ccdbcd22","Type":"ContainerStarted","Data":"3509b076b31c2e7a041c5a5a8b7331cc249ab1178622736196a521faac186172"} Oct 31 03:59:30.051822 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:30.051216 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 03:59:30.052721 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:30.052694 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" event={"ID":"dfb84fa0-c690-40cd-b19e-db6b1767bc34","Type":"ContainerStarted","Data":"92d4671acbcf81cfcb6c1806a0badb660251a81bccd043314e4d36fd39cbc271"} Oct 31 03:59:30.053708 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:30.053659 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp"] Oct 31 03:59:30.065174 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:30.060770 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab34e2db-vghsp"] Oct 31 03:59:30.161522 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:30.161466 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" podStartSLOduration=1.161446155 podStartE2EDuration="1.161446155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:30.160635848 +0000 UTC m=+1867.161261078" watchObservedRunningTime="2026-10-31 03:59:30.161446155 +0000 UTC m=+1867.162071412" Oct 31 03:59:31.066523 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:31.066480 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" event={"ID":"dfb84fa0-c690-40cd-b19e-db6b1767bc34","Type":"ContainerStarted","Data":"c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b"} Oct 31 03:59:31.066987 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:31.066566 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 03:59:31.155016 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:31.154981 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8474f84c-56a3-4e73-8936-edd0986ab7ed" path="/var/lib/kubelet/pods/8474f84c-56a3-4e73-8936-edd0986ab7ed/volumes" Oct 31 03:59:31.176978 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:31.176927 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" podStartSLOduration=2.176913432 podStartE2EDuration="2.176913432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:31.174661961 +0000 UTC m=+1868.175287385" watchObservedRunningTime="2026-10-31 03:59:31.176913432 +0000 UTC m=+1868.177538681" Oct 31 03:59:32.052551 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:32.052507 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr"] Oct 31 03:59:32.096015 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:32.095969 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv"] Oct 31 03:59:32.101957 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:32.101926 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4284dcb-r6dsr"] Oct 31 03:59:32.107529 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:32.107499 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17bdb58-rrcbv"] Oct 31 03:59:32.850337 ip-10-0-134-106 kubenswrapper[4426]: E1031 03:59:32.850296 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 03:59:33.135557 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:33.135473 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311a2b24-1212-488c-8f06-2629a7388dc3" path="/var/lib/kubelet/pods/311a2b24-1212-488c-8f06-2629a7388dc3/volumes" Oct 31 03:59:33.135997 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:33.135982 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f6a8a5-f521-43ee-9dbd-31aaacc00f24" path="/var/lib/kubelet/pods/91f6a8a5-f521-43ee-9dbd-31aaacc00f24/volumes" Oct 31 03:59:37.053149 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:37.053070 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd"] Oct 31 03:59:37.096604 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:37.096567 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221d193b-wkckd"] Oct 31 03:59:37.142151 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:37.141880 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79" path="/var/lib/kubelet/pods/7b6e2975-4231-4bb5-8ea1-ac39d5e8ab79/volumes" Oct 31 03:59:41.047078 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:41.047040 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p"] Oct 31 03:59:41.070493 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:41.070462 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 03:59:41.093372 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:41.093338 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea271435-9qk7p"] Oct 31 03:59:41.132556 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:41.132525 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6292b4c3-0129-43f1-b1f0-b78b4fd99ae0" path="/var/lib/kubelet/pods/6292b4c3-0129-43f1-b1f0-b78b4fd99ae0/volumes" Oct 31 03:59:42.081105 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:42.081068 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 03:59:55.053313 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:55.053283 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt"] Oct 31 03:59:55.098071 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:55.098034 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93800a09-lc9gt"] Oct 31 03:59:55.133570 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:55.133536 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1538000c-913d-4117-92b4-d2549581a428" path="/var/lib/kubelet/pods/1538000c-913d-4117-92b4-d2549581a428/volumes" Oct 31 03:59:57.049411 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:57.049375 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r"] Oct 31 03:59:57.094712 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:57.094666 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1c6bce-rgs5r"] Oct 31 03:59:57.133337 ip-10-0-134-106 kubenswrapper[4426]: I1031 03:59:57.133304 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c94b670-dab9-4ddc-af46-c6565dd0d16d" path="/var/lib/kubelet/pods/1c94b670-dab9-4ddc-af46-c6565dd0d16d/volumes" Oct 31 04:00:00.161673 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.161626 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw"] Oct 31 04:00:00.167762 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.167736 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7"] Oct 31 04:00:00.167908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.167883 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" Oct 31 04:00:00.172733 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.172707 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg"] Oct 31 04:00:00.172875 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.172818 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" Oct 31 04:00:00.177497 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.177471 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" Oct 31 04:00:00.208346 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.208315 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5"] Oct 31 04:00:00.213178 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.213153 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" Oct 31 04:00:00.213628 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.213606 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt"] Oct 31 04:00:00.219325 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.219301 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2"] Oct 31 04:00:00.219448 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.219406 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" Oct 31 04:00:00.224194 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.224169 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" Oct 31 04:00:00.224440 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.224429 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc"] Oct 31 04:00:00.228909 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.228893 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" Oct 31 04:00:00.229441 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.229429 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx"] Oct 31 04:00:00.234539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.234521 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" Oct 31 04:00:00.234617 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.234561 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv"] Oct 31 04:00:00.239903 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.239718 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw"] Oct 31 04:00:00.239903 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.239845 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" Oct 31 04:00:00.251895 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.251636 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:00:00.252266 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.252245 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:00:00.254041 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.253161 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:00:00.290201 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.260781 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7"] Oct 31 04:00:00.290201 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.279289 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg"] Oct 31 04:00:00.290201 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.287045 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5"] Oct 31 04:00:00.297254 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.297044 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt"] Oct 31 04:00:00.311885 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.311859 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv"] Oct 31 04:00:00.314363 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314275 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7trbd\" (UniqueName: \"kubernetes.io/projected/6340a3dd-2df2-4aaf-9a91-1a64798bed73-kube-api-access-7trbd\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg\" (UID: \"6340a3dd-2df2-4aaf-9a91-1a64798bed73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" Oct 31 04:00:00.314363 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314349 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5\" (UID: \"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" Oct 31 04:00:00.314571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314376 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx\" (UID: \"54338ebc-fe16-444d-9cfb-ff8537bb7c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" Oct 31 04:00:00.314571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314408 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87c25626-c789-442c-91f4-7d453fc1e26d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2\" (UID: \"87c25626-c789-442c-91f4-7d453fc1e26d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" Oct 31 04:00:00.314571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314445 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttj7\" (UniqueName: \"kubernetes.io/projected/1b647cae-10e2-44e2-8359-241944b5acfe-kube-api-access-zttj7\") pod \"maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw\" (UID: \"1b647cae-10e2-44e2-8359-241944b5acfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" Oct 31 04:00:00.314571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314494 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzl6\" (UniqueName: \"kubernetes.io/projected/c118d9fb-524e-4287-919e-fd9a760bc4bc-kube-api-access-smzl6\") pod \"maas-api-key-cleanup-29890320-9hcxc\" (UID: \"c118d9fb-524e-4287-919e-fd9a760bc4bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" Oct 31 04:00:00.314571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314517 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9hp\" (UniqueName: \"kubernetes.io/projected/87c25626-c789-442c-91f4-7d453fc1e26d-kube-api-access-tq9hp\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2\" (UID: \"87c25626-c789-442c-91f4-7d453fc1e26d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" Oct 31 04:00:00.314571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314558 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6340a3dd-2df2-4aaf-9a91-1a64798bed73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg\" (UID: \"6340a3dd-2df2-4aaf-9a91-1a64798bed73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" Oct 31 04:00:00.314969 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314604 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5jrg\" (UniqueName: \"kubernetes.io/projected/ec1cd5ae-d667-484f-b302-3680f74423d9-kube-api-access-z5jrg\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv\" (UID: \"ec1cd5ae-d667-484f-b302-3680f74423d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" Oct 31 04:00:00.314969 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314653 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b647cae-10e2-44e2-8359-241944b5acfe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw\" (UID: \"1b647cae-10e2-44e2-8359-241944b5acfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" Oct 31 04:00:00.314969 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314691 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29acde0-cc51-41a1-91c4-7522b9c4978c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt\" (UID: \"f29acde0-cc51-41a1-91c4-7522b9c4978c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" Oct 31 04:00:00.314969 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.314741 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1cd5ae-d667-484f-b302-3680f74423d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv\" (UID: \"ec1cd5ae-d667-484f-b302-3680f74423d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" Oct 31 04:00:00.315202 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.315022 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbtrf\" (UniqueName: \"kubernetes.io/projected/f29acde0-cc51-41a1-91c4-7522b9c4978c-kube-api-access-rbtrf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt\" (UID: \"f29acde0-cc51-41a1-91c4-7522b9c4978c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" Oct 31 04:00:00.315322 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.315262 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f779ce49-773a-4bc7-87db-be2271baa5d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7\" (UID: \"f779ce49-773a-4bc7-87db-be2271baa5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" Oct 31 04:00:00.315677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.315655 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvzs7\" (UniqueName: \"kubernetes.io/projected/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-kube-api-access-fvzs7\") pod \"maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx\" (UID: \"54338ebc-fe16-444d-9cfb-ff8537bb7c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" Oct 31 04:00:00.315916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.315897 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c118d9fb-524e-4287-919e-fd9a760bc4bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-9hcxc\" (UID: \"c118d9fb-524e-4287-919e-fd9a760bc4bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" Oct 31 04:00:00.316081 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.316066 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jscxb\" (UniqueName: \"kubernetes.io/projected/f779ce49-773a-4bc7-87db-be2271baa5d0-kube-api-access-jscxb\") pod \"maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7\" (UID: \"f779ce49-773a-4bc7-87db-be2271baa5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" Oct 31 04:00:00.316291 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.316274 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9lpl\" (UniqueName: \"kubernetes.io/projected/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-kube-api-access-w9lpl\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5\" (UID: \"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" Oct 31 04:00:00.320229 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.320208 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx"] Oct 31 04:00:00.335074 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.334780 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2"] Oct 31 04:00:00.348043 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.348015 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc"] Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.418070 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jscxb\" (UniqueName: \"kubernetes.io/projected/f779ce49-773a-4bc7-87db-be2271baa5d0-kube-api-access-jscxb\") pod \"maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7\" (UID: \"f779ce49-773a-4bc7-87db-be2271baa5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.418178 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9lpl\" (UniqueName: \"kubernetes.io/projected/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-kube-api-access-w9lpl\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5\" (UID: \"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.418456 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7trbd\" (UniqueName: \"kubernetes.io/projected/6340a3dd-2df2-4aaf-9a91-1a64798bed73-kube-api-access-7trbd\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg\" (UID: \"6340a3dd-2df2-4aaf-9a91-1a64798bed73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.418514 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5\" (UID: \"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.420631 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx\" (UID: \"54338ebc-fe16-444d-9cfb-ff8537bb7c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.420682 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87c25626-c789-442c-91f4-7d453fc1e26d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2\" (UID: \"87c25626-c789-442c-91f4-7d453fc1e26d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.420725 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttj7\" (UniqueName: \"kubernetes.io/projected/1b647cae-10e2-44e2-8359-241944b5acfe-kube-api-access-zttj7\") pod \"maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw\" (UID: \"1b647cae-10e2-44e2-8359-241944b5acfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.420781 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzl6\" (UniqueName: \"kubernetes.io/projected/c118d9fb-524e-4287-919e-fd9a760bc4bc-kube-api-access-smzl6\") pod \"maas-api-key-cleanup-29890320-9hcxc\" (UID: \"c118d9fb-524e-4287-919e-fd9a760bc4bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.420810 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9hp\" (UniqueName: \"kubernetes.io/projected/87c25626-c789-442c-91f4-7d453fc1e26d-kube-api-access-tq9hp\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2\" (UID: \"87c25626-c789-442c-91f4-7d453fc1e26d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.420847 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6340a3dd-2df2-4aaf-9a91-1a64798bed73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg\" (UID: \"6340a3dd-2df2-4aaf-9a91-1a64798bed73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.420894 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5jrg\" (UniqueName: \"kubernetes.io/projected/ec1cd5ae-d667-484f-b302-3680f74423d9-kube-api-access-z5jrg\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv\" (UID: \"ec1cd5ae-d667-484f-b302-3680f74423d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.420947 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b647cae-10e2-44e2-8359-241944b5acfe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw\" (UID: \"1b647cae-10e2-44e2-8359-241944b5acfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.420987 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29acde0-cc51-41a1-91c4-7522b9c4978c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt\" (UID: \"f29acde0-cc51-41a1-91c4-7522b9c4978c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.421038 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1cd5ae-d667-484f-b302-3680f74423d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv\" (UID: \"ec1cd5ae-d667-484f-b302-3680f74423d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.421106 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbtrf\" (UniqueName: \"kubernetes.io/projected/f29acde0-cc51-41a1-91c4-7522b9c4978c-kube-api-access-rbtrf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt\" (UID: \"f29acde0-cc51-41a1-91c4-7522b9c4978c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.421270 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f779ce49-773a-4bc7-87db-be2271baa5d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7\" (UID: \"f779ce49-773a-4bc7-87db-be2271baa5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.421396 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvzs7\" (UniqueName: \"kubernetes.io/projected/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-kube-api-access-fvzs7\") pod \"maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx\" (UID: \"54338ebc-fe16-444d-9cfb-ff8537bb7c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" Oct 31 04:00:00.421932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.421565 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c118d9fb-524e-4287-919e-fd9a760bc4bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-9hcxc\" (UID: \"c118d9fb-524e-4287-919e-fd9a760bc4bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" Oct 31 04:00:00.422863 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.422308 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6340a3dd-2df2-4aaf-9a91-1a64798bed73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg\" (UID: \"6340a3dd-2df2-4aaf-9a91-1a64798bed73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" Oct 31 04:00:00.422863 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.422424 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c118d9fb-524e-4287-919e-fd9a760bc4bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-9hcxc\" (UID: \"c118d9fb-524e-4287-919e-fd9a760bc4bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" Oct 31 04:00:00.423153 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.423102 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5\" (UID: \"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" Oct 31 04:00:00.423276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.423226 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b647cae-10e2-44e2-8359-241944b5acfe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw\" (UID: \"1b647cae-10e2-44e2-8359-241944b5acfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" Oct 31 04:00:00.423858 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.423830 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29acde0-cc51-41a1-91c4-7522b9c4978c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt\" (UID: \"f29acde0-cc51-41a1-91c4-7522b9c4978c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" Oct 31 04:00:00.423975 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.423832 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1cd5ae-d667-484f-b302-3680f74423d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv\" (UID: \"ec1cd5ae-d667-484f-b302-3680f74423d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" Oct 31 04:00:00.424043 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.423981 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f779ce49-773a-4bc7-87db-be2271baa5d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7\" (UID: \"f779ce49-773a-4bc7-87db-be2271baa5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" Oct 31 04:00:00.424541 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.424514 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87c25626-c789-442c-91f4-7d453fc1e26d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2\" (UID: \"87c25626-c789-442c-91f4-7d453fc1e26d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" Oct 31 04:00:00.424875 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.424850 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx\" (UID: \"54338ebc-fe16-444d-9cfb-ff8537bb7c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" Oct 31 04:00:00.480913 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.480417 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jscxb\" (UniqueName: \"kubernetes.io/projected/f779ce49-773a-4bc7-87db-be2271baa5d0-kube-api-access-jscxb\") pod \"maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7\" (UID: \"f779ce49-773a-4bc7-87db-be2271baa5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" Oct 31 04:00:00.481808 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.481774 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7trbd\" (UniqueName: \"kubernetes.io/projected/6340a3dd-2df2-4aaf-9a91-1a64798bed73-kube-api-access-7trbd\") pod \"maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg\" (UID: \"6340a3dd-2df2-4aaf-9a91-1a64798bed73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" Oct 31 04:00:00.482604 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.481862 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9lpl\" (UniqueName: \"kubernetes.io/projected/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-kube-api-access-w9lpl\") pod \"maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5\" (UID: \"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" Oct 31 04:00:00.482604 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.482253 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9hp\" (UniqueName: \"kubernetes.io/projected/87c25626-c789-442c-91f4-7d453fc1e26d-kube-api-access-tq9hp\") pod \"maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2\" (UID: \"87c25626-c789-442c-91f4-7d453fc1e26d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" Oct 31 04:00:00.482604 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.482388 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5jrg\" (UniqueName: \"kubernetes.io/projected/ec1cd5ae-d667-484f-b302-3680f74423d9-kube-api-access-z5jrg\") pod \"maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv\" (UID: \"ec1cd5ae-d667-484f-b302-3680f74423d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" Oct 31 04:00:00.483560 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.483538 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbtrf\" (UniqueName: \"kubernetes.io/projected/f29acde0-cc51-41a1-91c4-7522b9c4978c-kube-api-access-rbtrf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt\" (UID: \"f29acde0-cc51-41a1-91c4-7522b9c4978c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" Oct 31 04:00:00.483892 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.483868 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttj7\" (UniqueName: \"kubernetes.io/projected/1b647cae-10e2-44e2-8359-241944b5acfe-kube-api-access-zttj7\") pod \"maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw\" (UID: \"1b647cae-10e2-44e2-8359-241944b5acfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" Oct 31 04:00:00.484177 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.484160 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvzs7\" (UniqueName: \"kubernetes.io/projected/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-kube-api-access-fvzs7\") pod \"maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx\" (UID: \"54338ebc-fe16-444d-9cfb-ff8537bb7c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" Oct 31 04:00:00.485389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.485369 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzl6\" (UniqueName: \"kubernetes.io/projected/c118d9fb-524e-4287-919e-fd9a760bc4bc-kube-api-access-smzl6\") pod \"maas-api-key-cleanup-29890320-9hcxc\" (UID: \"c118d9fb-524e-4287-919e-fd9a760bc4bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" Oct 31 04:00:00.487320 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.487300 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" Oct 31 04:00:00.494920 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.494898 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" Oct 31 04:00:00.502230 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.502208 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" Oct 31 04:00:00.544533 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.544223 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" Oct 31 04:00:00.544533 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.544460 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" Oct 31 04:00:00.550625 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.550508 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" Oct 31 04:00:00.605802 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.605769 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" Oct 31 04:00:00.621791 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.621482 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" Oct 31 04:00:00.639489 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:00.638418 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" Oct 31 04:00:01.105862 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.105826 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7"] Oct 31 04:00:01.114954 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.114557 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw"] Oct 31 04:00:01.158115 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.158092 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg"] Oct 31 04:00:01.165391 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:00:01.165365 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6340a3dd_2df2_4aaf_9a91_1a64798bed73.slice/crio-75f66d3803538b860b400353500d333b503e2bfc852e8ca97b0f963bdb908689 WatchSource:0}: Error finding container 75f66d3803538b860b400353500d333b503e2bfc852e8ca97b0f963bdb908689: Status 404 returned error can't find the container with id 75f66d3803538b860b400353500d333b503e2bfc852e8ca97b0f963bdb908689 Oct 31 04:00:01.206885 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.206862 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv"] Oct 31 04:00:01.210815 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:00:01.210789 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1cd5ae_d667_484f_b302_3680f74423d9.slice/crio-9ef1a8dea9d3f31368654abebbc547f0a7bfd7875e48a931f57f269bb027af11 WatchSource:0}: Error finding container 9ef1a8dea9d3f31368654abebbc547f0a7bfd7875e48a931f57f269bb027af11: Status 404 returned error can't find the container with id 9ef1a8dea9d3f31368654abebbc547f0a7bfd7875e48a931f57f269bb027af11 Oct 31 04:00:01.221153 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.221111 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2"] Oct 31 04:00:01.234089 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.234034 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5"] Oct 31 04:00:01.269456 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.269265 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt"] Oct 31 04:00:01.322213 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:00:01.322069 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c25626_c789_442c_91f4_7d453fc1e26d.slice/crio-2313e64d0e691687fcd60197a256e876b47ff1e81d72029507a4e78214b4164f WatchSource:0}: Error finding container 2313e64d0e691687fcd60197a256e876b47ff1e81d72029507a4e78214b4164f: Status 404 returned error can't find the container with id 2313e64d0e691687fcd60197a256e876b47ff1e81d72029507a4e78214b4164f Oct 31 04:00:01.322713 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:00:01.322675 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67baf37_cce6_4ce0_ae7a_f23571fd3ea4.slice/crio-d779a4439e85c55fc90d05c7c4bc1d578cc64af60b44bcfc2465126034179089 WatchSource:0}: Error finding container d779a4439e85c55fc90d05c7c4bc1d578cc64af60b44bcfc2465126034179089: Status 404 returned error can't find the container with id d779a4439e85c55fc90d05c7c4bc1d578cc64af60b44bcfc2465126034179089 Oct 31 04:00:01.330877 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:00:01.330006 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29acde0_cc51_41a1_91c4_7522b9c4978c.slice/crio-954e36bb81524ac590b2c25598ed245c819b4fa9819ea5b031f0b712641f4ddb WatchSource:0}: Error finding container 954e36bb81524ac590b2c25598ed245c819b4fa9819ea5b031f0b712641f4ddb: Status 404 returned error can't find the container with id 954e36bb81524ac590b2c25598ed245c819b4fa9819ea5b031f0b712641f4ddb Oct 31 04:00:01.343383 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.343348 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc"] Oct 31 04:00:01.376531 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.376486 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx"] Oct 31 04:00:01.412698 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:00:01.412656 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc118d9fb_524e_4287_919e_fd9a760bc4bc.slice/crio-7f5b3cb1e738f90a1d9c6f3015602d31eff3dde72e3de6d561fae09dd28a3f06 WatchSource:0}: Error finding container 7f5b3cb1e738f90a1d9c6f3015602d31eff3dde72e3de6d561fae09dd28a3f06: Status 404 returned error can't find the container with id 7f5b3cb1e738f90a1d9c6f3015602d31eff3dde72e3de6d561fae09dd28a3f06 Oct 31 04:00:01.466979 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.466948 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" event={"ID":"ec1cd5ae-d667-484f-b302-3680f74423d9","Type":"ContainerStarted","Data":"9ef1a8dea9d3f31368654abebbc547f0a7bfd7875e48a931f57f269bb027af11"} Oct 31 04:00:01.468467 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.468439 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" event={"ID":"f779ce49-773a-4bc7-87db-be2271baa5d0","Type":"ContainerStarted","Data":"c4f709bb9eaab7bc336fbcff518698e72f3890d5c17c0cac06e6b029feb61968"} Oct 31 04:00:01.469913 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.469888 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" event={"ID":"f29acde0-cc51-41a1-91c4-7522b9c4978c","Type":"ContainerStarted","Data":"954e36bb81524ac590b2c25598ed245c819b4fa9819ea5b031f0b712641f4ddb"} Oct 31 04:00:01.471396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.471375 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" event={"ID":"54338ebc-fe16-444d-9cfb-ff8537bb7c2e","Type":"ContainerStarted","Data":"30ac0ebc89b91b8e75511f2c3b6ff318d88d072fe217eff26b4dafe6c3092c0c"} Oct 31 04:00:01.473069 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.473042 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" event={"ID":"6340a3dd-2df2-4aaf-9a91-1a64798bed73","Type":"ContainerStarted","Data":"75f66d3803538b860b400353500d333b503e2bfc852e8ca97b0f963bdb908689"} Oct 31 04:00:01.474720 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.474692 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" event={"ID":"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4","Type":"ContainerStarted","Data":"d779a4439e85c55fc90d05c7c4bc1d578cc64af60b44bcfc2465126034179089"} Oct 31 04:00:01.476524 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.476504 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" event={"ID":"c118d9fb-524e-4287-919e-fd9a760bc4bc","Type":"ContainerStarted","Data":"7f5b3cb1e738f90a1d9c6f3015602d31eff3dde72e3de6d561fae09dd28a3f06"} Oct 31 04:00:01.477754 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.477733 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" event={"ID":"87c25626-c789-442c-91f4-7d453fc1e26d","Type":"ContainerStarted","Data":"2313e64d0e691687fcd60197a256e876b47ff1e81d72029507a4e78214b4164f"} Oct 31 04:00:01.479029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:01.479007 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" event={"ID":"1b647cae-10e2-44e2-8359-241944b5acfe","Type":"ContainerStarted","Data":"e01bb0b1960fe31ce45cbded41771f2c0418219277244af18c7e5973a10dcb93"} Oct 31 04:00:02.524670 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.524627 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" event={"ID":"f29acde0-cc51-41a1-91c4-7522b9c4978c","Type":"ContainerStarted","Data":"d1039db66349ca22ffb695ca07839e0631e54551c3b8ac0884525f8ec14c272f"} Oct 31 04:00:02.527507 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.527429 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" event={"ID":"54338ebc-fe16-444d-9cfb-ff8537bb7c2e","Type":"ContainerStarted","Data":"31a749c30ffb83410c7f30dd551586fffb38436c3bf412e4f099d45ff3767e6b"} Oct 31 04:00:02.530662 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.529776 4426 generic.go:356] "Generic (PLEG): container finished" podID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerID="37d0c2667e27d220f197ebc61d797063f9d05c9d0e669e3d7927fe3005dd0ee9" exitCode=60 Oct 31 04:00:02.530662 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.530115 4426 scope.go:122] "RemoveContainer" containerID="37d0c2667e27d220f197ebc61d797063f9d05c9d0e669e3d7927fe3005dd0ee9" Oct 31 04:00:02.530662 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.530179 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" event={"ID":"6340a3dd-2df2-4aaf-9a91-1a64798bed73","Type":"ContainerDied","Data":"37d0c2667e27d220f197ebc61d797063f9d05c9d0e669e3d7927fe3005dd0ee9"} Oct 31 04:00:02.532424 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.532402 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" event={"ID":"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4","Type":"ContainerStarted","Data":"3ccfde7dc4ad70caf8b88fa630e5080326520492ec8fde21a692a6ef39205111"} Oct 31 04:00:02.534381 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.534329 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" event={"ID":"c118d9fb-524e-4287-919e-fd9a760bc4bc","Type":"ContainerStarted","Data":"c8c09c830ad1d1a215dcc49c2ad25927cfd64d37a1f2df10128b19fb4d0a6b9d"} Oct 31 04:00:02.537610 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.537585 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" event={"ID":"87c25626-c789-442c-91f4-7d453fc1e26d","Type":"ContainerStarted","Data":"35e9b78fe68d5d2359a7fd9d54a9dae26085888817ae19bc93632433aaeb7a6e"} Oct 31 04:00:02.539926 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.539897 4426 generic.go:356] "Generic (PLEG): container finished" podID="1b647cae-10e2-44e2-8359-241944b5acfe" containerID="a54069ebec240f7ef97ef9d64fd3b34782f4d0f0106daaf835a74c74871cac91" exitCode=60 Oct 31 04:00:02.540048 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.540026 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" event={"ID":"1b647cae-10e2-44e2-8359-241944b5acfe","Type":"ContainerDied","Data":"a54069ebec240f7ef97ef9d64fd3b34782f4d0f0106daaf835a74c74871cac91"} Oct 31 04:00:02.540250 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.540233 4426 scope.go:122] "RemoveContainer" containerID="a54069ebec240f7ef97ef9d64fd3b34782f4d0f0106daaf835a74c74871cac91" Oct 31 04:00:02.541926 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.541905 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" event={"ID":"ec1cd5ae-d667-484f-b302-3680f74423d9","Type":"ContainerStarted","Data":"6bf96fc3dd41f137c32b2e0f75efd05907d17637c35a727ad802fe3ed9ca0272"} Oct 31 04:00:02.546024 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.546001 4426 generic.go:356] "Generic (PLEG): container finished" podID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerID="1230cc79633829780784cb4fa4648ea000acda3d02ff3dd6e25b095b992bf7d1" exitCode=60 Oct 31 04:00:02.546174 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.546061 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" event={"ID":"f779ce49-773a-4bc7-87db-be2271baa5d0","Type":"ContainerDied","Data":"1230cc79633829780784cb4fa4648ea000acda3d02ff3dd6e25b095b992bf7d1"} Oct 31 04:00:02.546450 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.546429 4426 scope.go:122] "RemoveContainer" containerID="1230cc79633829780784cb4fa4648ea000acda3d02ff3dd6e25b095b992bf7d1" Oct 31 04:00:02.635661 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:02.635595 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" podStartSLOduration=2.635576902 podStartE2EDuration="2.635576902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:02.633496914 +0000 UTC m=+1899.634122144" watchObservedRunningTime="2026-10-31 04:00:02.635576902 +0000 UTC m=+1899.636202132" Oct 31 04:00:02.855715 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:02.855668 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:00:03.023423 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.023361 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" podStartSLOduration=3.023347478 podStartE2EDuration="3.023347478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.020115952 +0000 UTC m=+1900.020741179" watchObservedRunningTime="2026-10-31 04:00:03.023347478 +0000 UTC m=+1900.023972711" Oct 31 04:00:03.131039 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.130976 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" podStartSLOduration=3.130961616 podStartE2EDuration="3.130961616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.127862262 +0000 UTC m=+1900.128487491" watchObservedRunningTime="2026-10-31 04:00:03.130961616 +0000 UTC m=+1900.131586871" Oct 31 04:00:03.347561 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.347504 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" podStartSLOduration=3.347486686 podStartE2EDuration="3.347486686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.34659059 +0000 UTC m=+1900.347215819" watchObservedRunningTime="2026-10-31 04:00:03.347486686 +0000 UTC m=+1900.348111914" Oct 31 04:00:03.457067 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.457009 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" podStartSLOduration=3.456995372 podStartE2EDuration="3.456995372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.453480914 +0000 UTC m=+1900.454106141" watchObservedRunningTime="2026-10-31 04:00:03.456995372 +0000 UTC m=+1900.457620600" Oct 31 04:00:03.571091 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.571046 4426 generic.go:356] "Generic (PLEG): container finished" podID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerID="6bf96fc3dd41f137c32b2e0f75efd05907d17637c35a727ad802fe3ed9ca0272" exitCode=60 Oct 31 04:00:03.571660 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.571113 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" podStartSLOduration=3.57109667 podStartE2EDuration="3.57109667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.564842971 +0000 UTC m=+1900.565468200" watchObservedRunningTime="2026-10-31 04:00:03.57109667 +0000 UTC m=+1900.571721901" Oct 31 04:00:03.571660 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.571236 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" event={"ID":"ec1cd5ae-d667-484f-b302-3680f74423d9","Type":"ContainerDied","Data":"6bf96fc3dd41f137c32b2e0f75efd05907d17637c35a727ad802fe3ed9ca0272"} Oct 31 04:00:03.575091 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.575049 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" event={"ID":"f779ce49-773a-4bc7-87db-be2271baa5d0","Type":"ContainerStarted","Data":"7808b5ec08a9e38a9da3c2bcb4b279e13817f740583fdd3a16fb3036b259c544"} Oct 31 04:00:03.579814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.579780 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" event={"ID":"f29acde0-cc51-41a1-91c4-7522b9c4978c","Type":"ContainerDied","Data":"d1039db66349ca22ffb695ca07839e0631e54551c3b8ac0884525f8ec14c272f"} Oct 31 04:00:03.579814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.579788 4426 generic.go:356] "Generic (PLEG): container finished" podID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerID="d1039db66349ca22ffb695ca07839e0631e54551c3b8ac0884525f8ec14c272f" exitCode=60 Oct 31 04:00:03.581563 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.581544 4426 scope.go:122] "RemoveContainer" containerID="6bf96fc3dd41f137c32b2e0f75efd05907d17637c35a727ad802fe3ed9ca0272" Oct 31 04:00:03.581794 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.581770 4426 scope.go:122] "RemoveContainer" containerID="d1039db66349ca22ffb695ca07839e0631e54551c3b8ac0884525f8ec14c272f" Oct 31 04:00:03.582274 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.582246 4426 generic.go:356] "Generic (PLEG): container finished" podID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerID="31a749c30ffb83410c7f30dd551586fffb38436c3bf412e4f099d45ff3767e6b" exitCode=60 Oct 31 04:00:03.582802 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.582376 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" event={"ID":"54338ebc-fe16-444d-9cfb-ff8537bb7c2e","Type":"ContainerDied","Data":"31a749c30ffb83410c7f30dd551586fffb38436c3bf412e4f099d45ff3767e6b"} Oct 31 04:00:03.582802 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.582499 4426 scope.go:122] "RemoveContainer" containerID="31a749c30ffb83410c7f30dd551586fffb38436c3bf412e4f099d45ff3767e6b" Oct 31 04:00:03.586410 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.586384 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" event={"ID":"6340a3dd-2df2-4aaf-9a91-1a64798bed73","Type":"ContainerStarted","Data":"ef94d033c88694c9ed7e97fb7c6bcad59612abd32822ae02b5f80c1497e4573e"} Oct 31 04:00:03.588405 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.588380 4426 generic.go:356] "Generic (PLEG): container finished" podID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerID="3ccfde7dc4ad70caf8b88fa630e5080326520492ec8fde21a692a6ef39205111" exitCode=60 Oct 31 04:00:03.588512 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.588450 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" event={"ID":"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4","Type":"ContainerDied","Data":"3ccfde7dc4ad70caf8b88fa630e5080326520492ec8fde21a692a6ef39205111"} Oct 31 04:00:03.588683 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.588666 4426 scope.go:122] "RemoveContainer" containerID="3ccfde7dc4ad70caf8b88fa630e5080326520492ec8fde21a692a6ef39205111" Oct 31 04:00:03.590391 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.590369 4426 generic.go:356] "Generic (PLEG): container finished" podID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerID="c8c09c830ad1d1a215dcc49c2ad25927cfd64d37a1f2df10128b19fb4d0a6b9d" exitCode=60 Oct 31 04:00:03.590488 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.590460 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" event={"ID":"c118d9fb-524e-4287-919e-fd9a760bc4bc","Type":"ContainerDied","Data":"c8c09c830ad1d1a215dcc49c2ad25927cfd64d37a1f2df10128b19fb4d0a6b9d"} Oct 31 04:00:03.590669 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.590653 4426 scope.go:122] "RemoveContainer" containerID="c8c09c830ad1d1a215dcc49c2ad25927cfd64d37a1f2df10128b19fb4d0a6b9d" Oct 31 04:00:03.592344 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.592318 4426 generic.go:356] "Generic (PLEG): container finished" podID="87c25626-c789-442c-91f4-7d453fc1e26d" containerID="35e9b78fe68d5d2359a7fd9d54a9dae26085888817ae19bc93632433aaeb7a6e" exitCode=60 Oct 31 04:00:03.592431 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.592352 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" event={"ID":"87c25626-c789-442c-91f4-7d453fc1e26d","Type":"ContainerDied","Data":"35e9b78fe68d5d2359a7fd9d54a9dae26085888817ae19bc93632433aaeb7a6e"} Oct 31 04:00:03.592657 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.592639 4426 scope.go:122] "RemoveContainer" containerID="35e9b78fe68d5d2359a7fd9d54a9dae26085888817ae19bc93632433aaeb7a6e" Oct 31 04:00:03.594612 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.594586 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" event={"ID":"1b647cae-10e2-44e2-8359-241944b5acfe","Type":"ContainerStarted","Data":"59f45e7432347957f62ab10c9ed1b16ffbd56bcd7a3efb41c01e11cad6e2502f"} Oct 31 04:00:03.746762 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:03.746714 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" podStartSLOduration=3.746699117 podStartE2EDuration="3.746699117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.741024915 +0000 UTC m=+1900.741650142" watchObservedRunningTime="2026-10-31 04:00:03.746699117 +0000 UTC m=+1900.747324376" Oct 31 04:00:04.071003 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.070950 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" podStartSLOduration=4.070933387 podStartE2EDuration="4.070933387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:04.067712249 +0000 UTC m=+1901.068337478" watchObservedRunningTime="2026-10-31 04:00:04.070933387 +0000 UTC m=+1901.071558651" Oct 31 04:00:04.610727 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.610632 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" event={"ID":"54338ebc-fe16-444d-9cfb-ff8537bb7c2e","Type":"ContainerStarted","Data":"11e47c81600488abdb546aaa0b6ab07b0d654acb3996037d1482281cd4aea3a9"} Oct 31 04:00:04.615688 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.615652 4426 generic.go:356] "Generic (PLEG): container finished" podID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerID="ef94d033c88694c9ed7e97fb7c6bcad59612abd32822ae02b5f80c1497e4573e" exitCode=60 Oct 31 04:00:04.616513 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.616490 4426 scope.go:122] "RemoveContainer" containerID="ef94d033c88694c9ed7e97fb7c6bcad59612abd32822ae02b5f80c1497e4573e" Oct 31 04:00:04.628429 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:04.628383 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg_odh-ai-gateway-infra(6340a3dd-2df2-4aaf-9a91-1a64798bed73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" Oct 31 04:00:04.629875 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.629841 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" event={"ID":"6340a3dd-2df2-4aaf-9a91-1a64798bed73","Type":"ContainerDied","Data":"ef94d033c88694c9ed7e97fb7c6bcad59612abd32822ae02b5f80c1497e4573e"} Oct 31 04:00:04.643203 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.643172 4426 scope.go:122] "RemoveContainer" containerID="37d0c2667e27d220f197ebc61d797063f9d05c9d0e669e3d7927fe3005dd0ee9" Oct 31 04:00:04.658704 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.658663 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" event={"ID":"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4","Type":"ContainerStarted","Data":"74a1ef252b92481862bd3dc972ebd76f9784e92c3adaff463edc304bf88f6fc8"} Oct 31 04:00:04.664981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.664440 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" event={"ID":"c118d9fb-524e-4287-919e-fd9a760bc4bc","Type":"ContainerStarted","Data":"5aa9f2ed28308f9ba4139d9947760c9e5899f939fc641050d99035f9255deea2"} Oct 31 04:00:04.673513 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.669941 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" event={"ID":"87c25626-c789-442c-91f4-7d453fc1e26d","Type":"ContainerStarted","Data":"3798e32bd1b1a525c2dcb4b7cc3690d16cab946326502f167867f5f32ae6890a"} Oct 31 04:00:04.676021 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.674179 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" podStartSLOduration=4.674160803 podStartE2EDuration="4.674160803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:04.630315197 +0000 UTC m=+1901.630940428" watchObservedRunningTime="2026-10-31 04:00:04.674160803 +0000 UTC m=+1901.674786031" Oct 31 04:00:04.676021 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.675583 4426 generic.go:356] "Generic (PLEG): container finished" podID="1b647cae-10e2-44e2-8359-241944b5acfe" containerID="59f45e7432347957f62ab10c9ed1b16ffbd56bcd7a3efb41c01e11cad6e2502f" exitCode=60 Oct 31 04:00:04.676021 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.675673 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" event={"ID":"1b647cae-10e2-44e2-8359-241944b5acfe","Type":"ContainerDied","Data":"59f45e7432347957f62ab10c9ed1b16ffbd56bcd7a3efb41c01e11cad6e2502f"} Oct 31 04:00:04.676319 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.676061 4426 scope.go:122] "RemoveContainer" containerID="59f45e7432347957f62ab10c9ed1b16ffbd56bcd7a3efb41c01e11cad6e2502f" Oct 31 04:00:04.676516 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:04.676492 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw_odh-ai-gateway-infra(1b647cae-10e2-44e2-8359-241944b5acfe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" podUID="1b647cae-10e2-44e2-8359-241944b5acfe" Oct 31 04:00:04.679173 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.678688 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" event={"ID":"ec1cd5ae-d667-484f-b302-3680f74423d9","Type":"ContainerStarted","Data":"8e3c36d5a70acea4bb9938acec9c488c2543151e9bbe3d33a616110985997bd0"} Oct 31 04:00:04.681239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.681205 4426 generic.go:356] "Generic (PLEG): container finished" podID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerID="7808b5ec08a9e38a9da3c2bcb4b279e13817f740583fdd3a16fb3036b259c544" exitCode=60 Oct 31 04:00:04.681384 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.681323 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" event={"ID":"f779ce49-773a-4bc7-87db-be2271baa5d0","Type":"ContainerDied","Data":"7808b5ec08a9e38a9da3c2bcb4b279e13817f740583fdd3a16fb3036b259c544"} Oct 31 04:00:04.681630 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.681610 4426 scope.go:122] "RemoveContainer" containerID="7808b5ec08a9e38a9da3c2bcb4b279e13817f740583fdd3a16fb3036b259c544" Oct 31 04:00:04.682039 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:04.682015 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7_odh-ai-gateway-infra(f779ce49-773a-4bc7-87db-be2271baa5d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" Oct 31 04:00:04.683764 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.683737 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" event={"ID":"f29acde0-cc51-41a1-91c4-7522b9c4978c","Type":"ContainerStarted","Data":"bc0770113b3271322d209efd4befc6c844f723fb27e93931b17fe5519bf40f38"} Oct 31 04:00:04.686824 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.686804 4426 scope.go:122] "RemoveContainer" containerID="a54069ebec240f7ef97ef9d64fd3b34782f4d0f0106daaf835a74c74871cac91" Oct 31 04:00:04.704394 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:04.704367 4426 scope.go:122] "RemoveContainer" containerID="1230cc79633829780784cb4fa4648ea000acda3d02ff3dd6e25b095b992bf7d1" Oct 31 04:00:05.695784 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.695748 4426 generic.go:356] "Generic (PLEG): container finished" podID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerID="bc0770113b3271322d209efd4befc6c844f723fb27e93931b17fe5519bf40f38" exitCode=60 Oct 31 04:00:05.696286 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.695834 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" event={"ID":"f29acde0-cc51-41a1-91c4-7522b9c4978c","Type":"ContainerDied","Data":"bc0770113b3271322d209efd4befc6c844f723fb27e93931b17fe5519bf40f38"} Oct 31 04:00:05.696286 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.695943 4426 scope.go:122] "RemoveContainer" containerID="d1039db66349ca22ffb695ca07839e0631e54551c3b8ac0884525f8ec14c272f" Oct 31 04:00:05.696286 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.696235 4426 scope.go:122] "RemoveContainer" containerID="bc0770113b3271322d209efd4befc6c844f723fb27e93931b17fe5519bf40f38" Oct 31 04:00:05.696662 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:05.696637 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt_odh-ai-gateway-infra(f29acde0-cc51-41a1-91c4-7522b9c4978c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" Oct 31 04:00:05.698594 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.698571 4426 generic.go:356] "Generic (PLEG): container finished" podID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerID="11e47c81600488abdb546aaa0b6ab07b0d654acb3996037d1482281cd4aea3a9" exitCode=60 Oct 31 04:00:05.698716 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.698668 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" event={"ID":"54338ebc-fe16-444d-9cfb-ff8537bb7c2e","Type":"ContainerDied","Data":"11e47c81600488abdb546aaa0b6ab07b0d654acb3996037d1482281cd4aea3a9"} Oct 31 04:00:05.699297 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.699274 4426 scope.go:122] "RemoveContainer" containerID="11e47c81600488abdb546aaa0b6ab07b0d654acb3996037d1482281cd4aea3a9" Oct 31 04:00:05.699648 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:05.699619 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx_odh-ai-gateway-infra(54338ebc-fe16-444d-9cfb-ff8537bb7c2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" Oct 31 04:00:05.701236 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.701214 4426 scope.go:122] "RemoveContainer" containerID="ef94d033c88694c9ed7e97fb7c6bcad59612abd32822ae02b5f80c1497e4573e" Oct 31 04:00:05.701555 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:05.701526 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg_odh-ai-gateway-infra(6340a3dd-2df2-4aaf-9a91-1a64798bed73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" Oct 31 04:00:05.704209 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.704178 4426 generic.go:356] "Generic (PLEG): container finished" podID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerID="74a1ef252b92481862bd3dc972ebd76f9784e92c3adaff463edc304bf88f6fc8" exitCode=60 Oct 31 04:00:05.704311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.704265 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" event={"ID":"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4","Type":"ContainerDied","Data":"74a1ef252b92481862bd3dc972ebd76f9784e92c3adaff463edc304bf88f6fc8"} Oct 31 04:00:05.704511 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.704495 4426 scope.go:122] "RemoveContainer" containerID="74a1ef252b92481862bd3dc972ebd76f9784e92c3adaff463edc304bf88f6fc8" Oct 31 04:00:05.704832 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:05.704809 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5_odh-ai-gateway-infra(a67baf37-cce6-4ce0-ae7a-f23571fd3ea4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" Oct 31 04:00:05.707367 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.707345 4426 generic.go:356] "Generic (PLEG): container finished" podID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerID="5aa9f2ed28308f9ba4139d9947760c9e5899f939fc641050d99035f9255deea2" exitCode=60 Oct 31 04:00:05.707500 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.707467 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" event={"ID":"c118d9fb-524e-4287-919e-fd9a760bc4bc","Type":"ContainerDied","Data":"5aa9f2ed28308f9ba4139d9947760c9e5899f939fc641050d99035f9255deea2"} Oct 31 04:00:05.707722 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.707703 4426 scope.go:122] "RemoveContainer" containerID="5aa9f2ed28308f9ba4139d9947760c9e5899f939fc641050d99035f9255deea2" Oct 31 04:00:05.708048 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:05.708016 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-9hcxc_odh-ai-gateway-infra(c118d9fb-524e-4287-919e-fd9a760bc4bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" Oct 31 04:00:05.709931 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.709906 4426 generic.go:356] "Generic (PLEG): container finished" podID="87c25626-c789-442c-91f4-7d453fc1e26d" containerID="3798e32bd1b1a525c2dcb4b7cc3690d16cab946326502f167867f5f32ae6890a" exitCode=60 Oct 31 04:00:05.710025 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.709980 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" event={"ID":"87c25626-c789-442c-91f4-7d453fc1e26d","Type":"ContainerDied","Data":"3798e32bd1b1a525c2dcb4b7cc3690d16cab946326502f167867f5f32ae6890a"} Oct 31 04:00:05.710240 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.710222 4426 scope.go:122] "RemoveContainer" containerID="3798e32bd1b1a525c2dcb4b7cc3690d16cab946326502f167867f5f32ae6890a" Oct 31 04:00:05.710555 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:05.710529 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2_odh-ai-gateway-infra(87c25626-c789-442c-91f4-7d453fc1e26d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" Oct 31 04:00:05.712234 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.712215 4426 scope.go:122] "RemoveContainer" containerID="59f45e7432347957f62ab10c9ed1b16ffbd56bcd7a3efb41c01e11cad6e2502f" Oct 31 04:00:05.712488 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:05.712466 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw_odh-ai-gateway-infra(1b647cae-10e2-44e2-8359-241944b5acfe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" podUID="1b647cae-10e2-44e2-8359-241944b5acfe" Oct 31 04:00:05.714069 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.714043 4426 generic.go:356] "Generic (PLEG): container finished" podID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerID="8e3c36d5a70acea4bb9938acec9c488c2543151e9bbe3d33a616110985997bd0" exitCode=60 Oct 31 04:00:05.714192 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.714153 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" event={"ID":"ec1cd5ae-d667-484f-b302-3680f74423d9","Type":"ContainerDied","Data":"8e3c36d5a70acea4bb9938acec9c488c2543151e9bbe3d33a616110985997bd0"} Oct 31 04:00:05.714403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.714386 4426 scope.go:122] "RemoveContainer" containerID="8e3c36d5a70acea4bb9938acec9c488c2543151e9bbe3d33a616110985997bd0" Oct 31 04:00:05.714591 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.714572 4426 scope.go:122] "RemoveContainer" containerID="31a749c30ffb83410c7f30dd551586fffb38436c3bf412e4f099d45ff3767e6b" Oct 31 04:00:05.714673 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:05.714616 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv_odh-ai-gateway-infra(ec1cd5ae-d667-484f-b302-3680f74423d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" Oct 31 04:00:05.716683 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.716665 4426 scope.go:122] "RemoveContainer" containerID="7808b5ec08a9e38a9da3c2bcb4b279e13817f740583fdd3a16fb3036b259c544" Oct 31 04:00:05.716918 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:05.716897 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7_odh-ai-gateway-infra(f779ce49-773a-4bc7-87db-be2271baa5d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" Oct 31 04:00:05.731061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.731031 4426 scope.go:122] "RemoveContainer" containerID="3ccfde7dc4ad70caf8b88fa630e5080326520492ec8fde21a692a6ef39205111" Oct 31 04:00:05.746640 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.746616 4426 scope.go:122] "RemoveContainer" containerID="c8c09c830ad1d1a215dcc49c2ad25927cfd64d37a1f2df10128b19fb4d0a6b9d" Oct 31 04:00:05.762175 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.762116 4426 scope.go:122] "RemoveContainer" containerID="35e9b78fe68d5d2359a7fd9d54a9dae26085888817ae19bc93632433aaeb7a6e" Oct 31 04:00:05.778378 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:05.778348 4426 scope.go:122] "RemoveContainer" containerID="6bf96fc3dd41f137c32b2e0f75efd05907d17637c35a727ad802fe3ed9ca0272" Oct 31 04:00:06.733547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:06.733516 4426 scope.go:122] "RemoveContainer" containerID="8e3c36d5a70acea4bb9938acec9c488c2543151e9bbe3d33a616110985997bd0" Oct 31 04:00:06.734013 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:06.733952 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv_odh-ai-gateway-infra(ec1cd5ae-d667-484f-b302-3680f74423d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" Oct 31 04:00:06.735736 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:06.735710 4426 scope.go:122] "RemoveContainer" containerID="bc0770113b3271322d209efd4befc6c844f723fb27e93931b17fe5519bf40f38" Oct 31 04:00:06.736101 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:06.736079 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt_odh-ai-gateway-infra(f29acde0-cc51-41a1-91c4-7522b9c4978c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" Oct 31 04:00:06.737774 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:06.737752 4426 scope.go:122] "RemoveContainer" containerID="11e47c81600488abdb546aaa0b6ab07b0d654acb3996037d1482281cd4aea3a9" Oct 31 04:00:06.738184 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:06.738121 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx_odh-ai-gateway-infra(54338ebc-fe16-444d-9cfb-ff8537bb7c2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" Oct 31 04:00:06.739738 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:06.739717 4426 scope.go:122] "RemoveContainer" containerID="74a1ef252b92481862bd3dc972ebd76f9784e92c3adaff463edc304bf88f6fc8" Oct 31 04:00:06.740082 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:06.740056 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5_odh-ai-gateway-infra(a67baf37-cce6-4ce0-ae7a-f23571fd3ea4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" Oct 31 04:00:06.741717 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:06.741696 4426 scope.go:122] "RemoveContainer" containerID="5aa9f2ed28308f9ba4139d9947760c9e5899f939fc641050d99035f9255deea2" Oct 31 04:00:06.742024 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:06.741994 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-9hcxc_odh-ai-gateway-infra(c118d9fb-524e-4287-919e-fd9a760bc4bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" Oct 31 04:00:06.743672 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:06.743652 4426 scope.go:122] "RemoveContainer" containerID="3798e32bd1b1a525c2dcb4b7cc3690d16cab946326502f167867f5f32ae6890a" Oct 31 04:00:06.743948 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:06.743929 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2_odh-ai-gateway-infra(87c25626-c789-442c-91f4-7d453fc1e26d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" Oct 31 04:00:08.483660 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:08.483625 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw"] Oct 31 04:00:08.769598 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:08.769553 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" event={"ID":"1b647cae-10e2-44e2-8359-241944b5acfe","Type":"ContainerDied","Data":"e01bb0b1960fe31ce45cbded41771f2c0418219277244af18c7e5973a10dcb93"} Oct 31 04:00:08.769786 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:08.769609 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e01bb0b1960fe31ce45cbded41771f2c0418219277244af18c7e5973a10dcb93" Oct 31 04:00:08.788849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:08.788815 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" Oct 31 04:00:08.893161 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:08.886786 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b647cae-10e2-44e2-8359-241944b5acfe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b647cae-10e2-44e2-8359-241944b5acfe-maas-api-service-ca\") pod \"1b647cae-10e2-44e2-8359-241944b5acfe\" (UID: \"1b647cae-10e2-44e2-8359-241944b5acfe\") " Oct 31 04:00:08.893161 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:08.887144 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b647cae-10e2-44e2-8359-241944b5acfe-kube-api-access-zttj7\" (UniqueName: \"kubernetes.io/projected/1b647cae-10e2-44e2-8359-241944b5acfe-kube-api-access-zttj7\") pod \"1b647cae-10e2-44e2-8359-241944b5acfe\" (UID: \"1b647cae-10e2-44e2-8359-241944b5acfe\") " Oct 31 04:00:08.893161 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:08.888397 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b647cae-10e2-44e2-8359-241944b5acfe-maas-api-service-ca" pod "1b647cae-10e2-44e2-8359-241944b5acfe" (UID: "1b647cae-10e2-44e2-8359-241944b5acfe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:08.905144 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:08.902692 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b647cae-10e2-44e2-8359-241944b5acfe-kube-api-access-zttj7" pod "1b647cae-10e2-44e2-8359-241944b5acfe" (UID: "1b647cae-10e2-44e2-8359-241944b5acfe"). InnerVolumeSpecName "kube-api-access-zttj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:08.990889 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:08.990846 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b647cae-10e2-44e2-8359-241944b5acfe-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:08.990889 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:08.990888 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zttj7\" (UniqueName: \"kubernetes.io/projected/1b647cae-10e2-44e2-8359-241944b5acfe-kube-api-access-zttj7\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:09.778807 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:09.778777 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw" Oct 31 04:00:09.941032 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:09.940987 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw"] Oct 31 04:00:09.987296 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:09.987255 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-6c89303e-29890320-zl6fw"] Oct 31 04:00:10.401477 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:10.401439 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz"] Oct 31 04:00:10.417150 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:10.416918 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c89303e-gv8pz"] Oct 31 04:00:11.135650 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:11.135619 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b647cae-10e2-44e2-8359-241944b5acfe" path="/var/lib/kubelet/pods/1b647cae-10e2-44e2-8359-241944b5acfe/volumes" Oct 31 04:00:11.136249 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:11.136231 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b62c0e50-46ac-4d91-9976-5471764fc6ef" path="/var/lib/kubelet/pods/b62c0e50-46ac-4d91-9976-5471764fc6ef/volumes" Oct 31 04:00:16.122516 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:16.122472 4426 scope.go:122] "RemoveContainer" containerID="7808b5ec08a9e38a9da3c2bcb4b279e13817f740583fdd3a16fb3036b259c544" Oct 31 04:00:16.123046 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:16.123023 4426 scope.go:122] "RemoveContainer" containerID="11e47c81600488abdb546aaa0b6ab07b0d654acb3996037d1482281cd4aea3a9" Oct 31 04:00:16.881452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:16.881362 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" event={"ID":"f779ce49-773a-4bc7-87db-be2271baa5d0","Type":"ContainerStarted","Data":"62165c871e27e63c91f8bd72f7552e523f8537b618c3be59836aee1d2c0f575a"} Oct 31 04:00:16.883864 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:16.883833 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" event={"ID":"54338ebc-fe16-444d-9cfb-ff8537bb7c2e","Type":"ContainerStarted","Data":"3fdd98eb4c6a21a569320234f34909fb2b4e2c1a4e78ec62d053b55356dfed3c"} Oct 31 04:00:17.122574 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.122544 4426 scope.go:122] "RemoveContainer" containerID="bc0770113b3271322d209efd4befc6c844f723fb27e93931b17fe5519bf40f38" Oct 31 04:00:17.123017 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.122752 4426 scope.go:122] "RemoveContainer" containerID="74a1ef252b92481862bd3dc972ebd76f9784e92c3adaff463edc304bf88f6fc8" Oct 31 04:00:17.125053 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.125021 4426 scope.go:122] "RemoveContainer" containerID="ef94d033c88694c9ed7e97fb7c6bcad59612abd32822ae02b5f80c1497e4573e" Oct 31 04:00:17.126521 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.126501 4426 scope.go:122] "RemoveContainer" containerID="8e3c36d5a70acea4bb9938acec9c488c2543151e9bbe3d33a616110985997bd0" Oct 31 04:00:17.127202 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.127183 4426 scope.go:122] "RemoveContainer" containerID="5aa9f2ed28308f9ba4139d9947760c9e5899f939fc641050d99035f9255deea2" Oct 31 04:00:17.895296 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.895258 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" event={"ID":"6340a3dd-2df2-4aaf-9a91-1a64798bed73","Type":"ContainerStarted","Data":"75515b7e9ea8bea2dbcdd320f98cafb0618ee7b1449d7a6b57e1d737a8132a8d"} Oct 31 04:00:17.897444 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.897412 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" event={"ID":"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4","Type":"ContainerStarted","Data":"070168500a2a29d1f4ce77e392ab69de34d125beed4adfe14cca56c9be5ac9ab"} Oct 31 04:00:17.899517 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.899490 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" event={"ID":"c118d9fb-524e-4287-919e-fd9a760bc4bc","Type":"ContainerStarted","Data":"9ecada6226f1ba1726ca26f9754842edfdd3945d8668caaabf77e2413c404f7a"} Oct 31 04:00:17.901561 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.901534 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" event={"ID":"ec1cd5ae-d667-484f-b302-3680f74423d9","Type":"ContainerStarted","Data":"62e3ab77972b889eda0e53eb6ffd495820407e113df38d43d5189ff42e99f93c"} Oct 31 04:00:17.903543 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.903520 4426 generic.go:356] "Generic (PLEG): container finished" podID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerID="62165c871e27e63c91f8bd72f7552e523f8537b618c3be59836aee1d2c0f575a" exitCode=60 Oct 31 04:00:17.903661 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.903589 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" event={"ID":"f779ce49-773a-4bc7-87db-be2271baa5d0","Type":"ContainerDied","Data":"62165c871e27e63c91f8bd72f7552e523f8537b618c3be59836aee1d2c0f575a"} Oct 31 04:00:17.903661 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.903634 4426 scope.go:122] "RemoveContainer" containerID="7808b5ec08a9e38a9da3c2bcb4b279e13817f740583fdd3a16fb3036b259c544" Oct 31 04:00:17.903925 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.903906 4426 scope.go:122] "RemoveContainer" containerID="62165c871e27e63c91f8bd72f7552e523f8537b618c3be59836aee1d2c0f575a" Oct 31 04:00:17.904235 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:17.904210 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7_odh-ai-gateway-infra(f779ce49-773a-4bc7-87db-be2271baa5d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" Oct 31 04:00:17.905960 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.905935 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" event={"ID":"f29acde0-cc51-41a1-91c4-7522b9c4978c","Type":"ContainerStarted","Data":"309799667fdccdd90bdfbe042eb3945e29089420044ed7c1d65fbf82584c12ab"} Oct 31 04:00:17.907922 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.907897 4426 generic.go:356] "Generic (PLEG): container finished" podID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerID="3fdd98eb4c6a21a569320234f34909fb2b4e2c1a4e78ec62d053b55356dfed3c" exitCode=60 Oct 31 04:00:17.907922 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.907913 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" event={"ID":"54338ebc-fe16-444d-9cfb-ff8537bb7c2e","Type":"ContainerDied","Data":"3fdd98eb4c6a21a569320234f34909fb2b4e2c1a4e78ec62d053b55356dfed3c"} Oct 31 04:00:17.908242 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.908226 4426 scope.go:122] "RemoveContainer" containerID="3fdd98eb4c6a21a569320234f34909fb2b4e2c1a4e78ec62d053b55356dfed3c" Oct 31 04:00:17.908465 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:17.908449 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx_odh-ai-gateway-infra(54338ebc-fe16-444d-9cfb-ff8537bb7c2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" Oct 31 04:00:17.921762 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:17.921653 4426 scope.go:122] "RemoveContainer" containerID="11e47c81600488abdb546aaa0b6ab07b0d654acb3996037d1482281cd4aea3a9" Oct 31 04:00:18.008717 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.008636 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7"] Oct 31 04:00:18.119248 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.119208 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx"] Oct 31 04:00:18.122708 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.122678 4426 scope.go:122] "RemoveContainer" containerID="3798e32bd1b1a525c2dcb4b7cc3690d16cab946326502f167867f5f32ae6890a" Oct 31 04:00:18.924603 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.924570 4426 generic.go:356] "Generic (PLEG): container finished" podID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerID="75515b7e9ea8bea2dbcdd320f98cafb0618ee7b1449d7a6b57e1d737a8132a8d" exitCode=60 Oct 31 04:00:18.924745 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.924649 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" event={"ID":"6340a3dd-2df2-4aaf-9a91-1a64798bed73","Type":"ContainerDied","Data":"75515b7e9ea8bea2dbcdd320f98cafb0618ee7b1449d7a6b57e1d737a8132a8d"} Oct 31 04:00:18.924745 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.924721 4426 scope.go:122] "RemoveContainer" containerID="ef94d033c88694c9ed7e97fb7c6bcad59612abd32822ae02b5f80c1497e4573e" Oct 31 04:00:18.925222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.925199 4426 scope.go:122] "RemoveContainer" containerID="75515b7e9ea8bea2dbcdd320f98cafb0618ee7b1449d7a6b57e1d737a8132a8d" Oct 31 04:00:18.925659 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:18.925629 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg_odh-ai-gateway-infra(6340a3dd-2df2-4aaf-9a91-1a64798bed73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" Oct 31 04:00:18.927535 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.927514 4426 generic.go:356] "Generic (PLEG): container finished" podID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerID="070168500a2a29d1f4ce77e392ab69de34d125beed4adfe14cca56c9be5ac9ab" exitCode=60 Oct 31 04:00:18.927644 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.927570 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" event={"ID":"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4","Type":"ContainerDied","Data":"070168500a2a29d1f4ce77e392ab69de34d125beed4adfe14cca56c9be5ac9ab"} Oct 31 04:00:18.928042 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.928023 4426 scope.go:122] "RemoveContainer" containerID="070168500a2a29d1f4ce77e392ab69de34d125beed4adfe14cca56c9be5ac9ab" Oct 31 04:00:18.928483 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:18.928457 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5_odh-ai-gateway-infra(a67baf37-cce6-4ce0-ae7a-f23571fd3ea4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" Oct 31 04:00:18.930066 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.930043 4426 generic.go:356] "Generic (PLEG): container finished" podID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerID="9ecada6226f1ba1726ca26f9754842edfdd3945d8668caaabf77e2413c404f7a" exitCode=60 Oct 31 04:00:18.930537 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.930115 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" event={"ID":"c118d9fb-524e-4287-919e-fd9a760bc4bc","Type":"ContainerDied","Data":"9ecada6226f1ba1726ca26f9754842edfdd3945d8668caaabf77e2413c404f7a"} Oct 31 04:00:18.930615 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.930593 4426 scope.go:122] "RemoveContainer" containerID="9ecada6226f1ba1726ca26f9754842edfdd3945d8668caaabf77e2413c404f7a" Oct 31 04:00:18.931047 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:18.931010 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-9hcxc_odh-ai-gateway-infra(c118d9fb-524e-4287-919e-fd9a760bc4bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" Oct 31 04:00:18.932859 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.932834 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" event={"ID":"87c25626-c789-442c-91f4-7d453fc1e26d","Type":"ContainerStarted","Data":"fecd3126104a80d04a96948ca6b509d5be4b91001df5fc2b99ca89859ececcec"} Oct 31 04:00:18.935488 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.935467 4426 generic.go:356] "Generic (PLEG): container finished" podID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerID="62e3ab77972b889eda0e53eb6ffd495820407e113df38d43d5189ff42e99f93c" exitCode=60 Oct 31 04:00:18.935605 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.935540 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" event={"ID":"ec1cd5ae-d667-484f-b302-3680f74423d9","Type":"ContainerDied","Data":"62e3ab77972b889eda0e53eb6ffd495820407e113df38d43d5189ff42e99f93c"} Oct 31 04:00:18.935968 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.935947 4426 scope.go:122] "RemoveContainer" containerID="62e3ab77972b889eda0e53eb6ffd495820407e113df38d43d5189ff42e99f93c" Oct 31 04:00:18.936366 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:18.936342 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv_odh-ai-gateway-infra(ec1cd5ae-d667-484f-b302-3680f74423d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" Oct 31 04:00:18.951147 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.951047 4426 generic.go:356] "Generic (PLEG): container finished" podID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerID="309799667fdccdd90bdfbe042eb3945e29089420044ed7c1d65fbf82584c12ab" exitCode=60 Oct 31 04:00:18.951378 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.951114 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" event={"ID":"f29acde0-cc51-41a1-91c4-7522b9c4978c","Type":"ContainerDied","Data":"309799667fdccdd90bdfbe042eb3945e29089420044ed7c1d65fbf82584c12ab"} Oct 31 04:00:18.951886 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.951866 4426 scope.go:122] "RemoveContainer" containerID="309799667fdccdd90bdfbe042eb3945e29089420044ed7c1d65fbf82584c12ab" Oct 31 04:00:18.952333 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:18.952306 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt_odh-ai-gateway-infra(f29acde0-cc51-41a1-91c4-7522b9c4978c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" Oct 31 04:00:18.953694 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.953420 4426 scope.go:122] "RemoveContainer" containerID="74a1ef252b92481862bd3dc972ebd76f9784e92c3adaff463edc304bf88f6fc8" Oct 31 04:00:18.974563 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.974438 4426 scope.go:122] "RemoveContainer" containerID="5aa9f2ed28308f9ba4139d9947760c9e5899f939fc641050d99035f9255deea2" Oct 31 04:00:18.992612 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:18.992254 4426 scope.go:122] "RemoveContainer" containerID="8e3c36d5a70acea4bb9938acec9c488c2543151e9bbe3d33a616110985997bd0" Oct 31 04:00:19.012055 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.010640 4426 scope.go:122] "RemoveContainer" containerID="bc0770113b3271322d209efd4befc6c844f723fb27e93931b17fe5519bf40f38" Oct 31 04:00:19.059751 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.046141 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg"] Oct 31 04:00:19.151523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.151418 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" Oct 31 04:00:19.161188 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.161155 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5"] Oct 31 04:00:19.238380 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.228931 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-maas-api-service-ca\") pod \"54338ebc-fe16-444d-9cfb-ff8537bb7c2e\" (UID: \"54338ebc-fe16-444d-9cfb-ff8537bb7c2e\") " Oct 31 04:00:19.238380 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.229242 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-kube-api-access-fvzs7\" (UniqueName: \"kubernetes.io/projected/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-kube-api-access-fvzs7\") pod \"54338ebc-fe16-444d-9cfb-ff8537bb7c2e\" (UID: \"54338ebc-fe16-444d-9cfb-ff8537bb7c2e\") " Oct 31 04:00:19.238380 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.231486 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-maas-api-service-ca" pod "54338ebc-fe16-444d-9cfb-ff8537bb7c2e" (UID: "54338ebc-fe16-444d-9cfb-ff8537bb7c2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:19.249101 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.239820 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-kube-api-access-fvzs7" pod "54338ebc-fe16-444d-9cfb-ff8537bb7c2e" (UID: "54338ebc-fe16-444d-9cfb-ff8537bb7c2e"). InnerVolumeSpecName "kube-api-access-fvzs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:19.304026 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.303995 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" Oct 31 04:00:19.332347 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.332053 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvzs7\" (UniqueName: \"kubernetes.io/projected/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-kube-api-access-fvzs7\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.332347 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.332091 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54338ebc-fe16-444d-9cfb-ff8537bb7c2e-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.434945 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.434843 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f779ce49-773a-4bc7-87db-be2271baa5d0-kube-api-access-jscxb\" (UniqueName: \"kubernetes.io/projected/f779ce49-773a-4bc7-87db-be2271baa5d0-kube-api-access-jscxb\") pod \"f779ce49-773a-4bc7-87db-be2271baa5d0\" (UID: \"f779ce49-773a-4bc7-87db-be2271baa5d0\") " Oct 31 04:00:19.435110 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.435064 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f779ce49-773a-4bc7-87db-be2271baa5d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f779ce49-773a-4bc7-87db-be2271baa5d0-maas-api-service-ca\") pod \"f779ce49-773a-4bc7-87db-be2271baa5d0\" (UID: \"f779ce49-773a-4bc7-87db-be2271baa5d0\") " Oct 31 04:00:19.435558 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.435508 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f779ce49-773a-4bc7-87db-be2271baa5d0-maas-api-service-ca" pod "f779ce49-773a-4bc7-87db-be2271baa5d0" (UID: "f779ce49-773a-4bc7-87db-be2271baa5d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:19.436633 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.436603 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f779ce49-773a-4bc7-87db-be2271baa5d0-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.437415 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.437390 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f779ce49-773a-4bc7-87db-be2271baa5d0-kube-api-access-jscxb" pod "f779ce49-773a-4bc7-87db-be2271baa5d0" (UID: "f779ce49-773a-4bc7-87db-be2271baa5d0"). InnerVolumeSpecName "kube-api-access-jscxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:19.467967 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.467935 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv"] Oct 31 04:00:19.538868 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.538823 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jscxb\" (UniqueName: \"kubernetes.io/projected/f779ce49-773a-4bc7-87db-be2271baa5d0-kube-api-access-jscxb\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.563911 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.563870 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc"] Oct 31 04:00:19.734860 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.734764 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt"] Oct 31 04:00:19.946748 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.946712 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:00:19.979791 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.979748 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" event={"ID":"54338ebc-fe16-444d-9cfb-ff8537bb7c2e","Type":"ContainerDied","Data":"30ac0ebc89b91b8e75511f2c3b6ff318d88d072fe217eff26b4dafe6c3092c0c"} Oct 31 04:00:19.980112 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.980092 4426 scope.go:122] "RemoveContainer" containerID="3fdd98eb4c6a21a569320234f34909fb2b4e2c1a4e78ec62d053b55356dfed3c" Oct 31 04:00:19.980409 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:19.980390 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx" Oct 31 04:00:20.030214 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.028953 4426 generic.go:356] "Generic (PLEG): container finished" podID="87c25626-c789-442c-91f4-7d453fc1e26d" containerID="fecd3126104a80d04a96948ca6b509d5be4b91001df5fc2b99ca89859ececcec" exitCode=60 Oct 31 04:00:20.030214 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.029150 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" event={"ID":"87c25626-c789-442c-91f4-7d453fc1e26d","Type":"ContainerDied","Data":"fecd3126104a80d04a96948ca6b509d5be4b91001df5fc2b99ca89859ececcec"} Oct 31 04:00:20.030214 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.029681 4426 scope.go:122] "RemoveContainer" containerID="fecd3126104a80d04a96948ca6b509d5be4b91001df5fc2b99ca89859ececcec" Oct 31 04:00:20.032984 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:20.032579 4426 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2_odh-ai-gateway-infra(87c25626-c789-442c-91f4-7d453fc1e26d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" Oct 31 04:00:20.038172 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.037935 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" event={"ID":"f779ce49-773a-4bc7-87db-be2271baa5d0","Type":"ContainerDied","Data":"c4f709bb9eaab7bc336fbcff518698e72f3890d5c17c0cac06e6b029feb61968"} Oct 31 04:00:20.038172 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.038077 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7" Oct 31 04:00:20.206077 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.206025 4426 scope.go:122] "RemoveContainer" containerID="3798e32bd1b1a525c2dcb4b7cc3690d16cab946326502f167867f5f32ae6890a" Oct 31 04:00:20.499755 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.498925 4426 scope.go:122] "RemoveContainer" containerID="62165c871e27e63c91f8bd72f7552e523f8537b618c3be59836aee1d2c0f575a" Oct 31 04:00:20.511510 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.511381 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2"] Oct 31 04:00:20.739156 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.736517 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" Oct 31 04:00:20.811254 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.809299 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-maas-api-service-ca\") pod \"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4\" (UID: \"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4\") " Oct 31 04:00:20.811254 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.809496 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-kube-api-access-w9lpl\" (UniqueName: \"kubernetes.io/projected/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-kube-api-access-w9lpl\") pod \"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4\" (UID: \"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4\") " Oct 31 04:00:20.820718 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.820681 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-maas-api-service-ca" pod "a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" (UID: "a67baf37-cce6-4ce0-ae7a-f23571fd3ea4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:20.916189 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.913272 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:21.009384 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:20.975898 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-kube-api-access-w9lpl" pod "a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" (UID: "a67baf37-cce6-4ce0-ae7a-f23571fd3ea4"). InnerVolumeSpecName "kube-api-access-w9lpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:21.009384 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.006502 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx"] Oct 31 04:00:21.016912 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.014787 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9lpl\" (UniqueName: \"kubernetes.io/projected/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4-kube-api-access-w9lpl\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:21.041084 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.041040 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-22e20532-29890320-bbxvx"] Oct 31 04:00:21.193219 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.091187 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7"] Oct 31 04:00:21.193219 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.149891 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" Oct 31 04:00:21.477623 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.463550 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" path="/var/lib/kubelet/pods/54338ebc-fe16-444d-9cfb-ff8537bb7c2e/volumes" Oct 31 04:00:21.477623 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.464610 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5" event={"ID":"a67baf37-cce6-4ce0-ae7a-f23571fd3ea4","Type":"ContainerDied","Data":"d779a4439e85c55fc90d05c7c4bc1d578cc64af60b44bcfc2465126034179089"} Oct 31 04:00:21.477623 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.464685 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eb5ce91f-29890320-kggl7"] Oct 31 04:00:21.477623 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.464708 4426 scope.go:122] "RemoveContainer" containerID="070168500a2a29d1f4ce77e392ab69de34d125beed4adfe14cca56c9be5ac9ab" Oct 31 04:00:21.704622 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.704539 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2"] Oct 31 04:00:21.746198 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.719839 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" Oct 31 04:00:21.746198 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.719945 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb5ce91f-ld8s2"] Oct 31 04:00:21.756147 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.749848 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1cd5ae-d667-484f-b302-3680f74423d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1cd5ae-d667-484f-b302-3680f74423d9-maas-api-service-ca\") pod \"ec1cd5ae-d667-484f-b302-3680f74423d9\" (UID: \"ec1cd5ae-d667-484f-b302-3680f74423d9\") " Oct 31 04:00:21.756147 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.749974 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1cd5ae-d667-484f-b302-3680f74423d9-kube-api-access-z5jrg\" (UniqueName: \"kubernetes.io/projected/ec1cd5ae-d667-484f-b302-3680f74423d9-kube-api-access-z5jrg\") pod \"ec1cd5ae-d667-484f-b302-3680f74423d9\" (UID: \"ec1cd5ae-d667-484f-b302-3680f74423d9\") " Oct 31 04:00:21.778709 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.778629 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1cd5ae-d667-484f-b302-3680f74423d9-maas-api-service-ca" pod "ec1cd5ae-d667-484f-b302-3680f74423d9" (UID: "ec1cd5ae-d667-484f-b302-3680f74423d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:21.872354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.862663 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1cd5ae-d667-484f-b302-3680f74423d9-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:21.885429 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.885176 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1cd5ae-d667-484f-b302-3680f74423d9-kube-api-access-z5jrg" pod "ec1cd5ae-d667-484f-b302-3680f74423d9" (UID: "ec1cd5ae-d667-484f-b302-3680f74423d9"). InnerVolumeSpecName "kube-api-access-z5jrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:21.885788 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.885763 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" Oct 31 04:00:21.898598 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.898558 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" Oct 31 04:00:21.916189 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.910566 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" Oct 31 04:00:22.003201 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.998194 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29acde0-cc51-41a1-91c4-7522b9c4978c-kube-api-access-rbtrf\" (UniqueName: \"kubernetes.io/projected/f29acde0-cc51-41a1-91c4-7522b9c4978c-kube-api-access-rbtrf\") pod \"f29acde0-cc51-41a1-91c4-7522b9c4978c\" (UID: \"f29acde0-cc51-41a1-91c4-7522b9c4978c\") " Oct 31 04:00:22.003201 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:21.998925 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c118d9fb-524e-4287-919e-fd9a760bc4bc-kube-api-access-smzl6\" (UniqueName: \"kubernetes.io/projected/c118d9fb-524e-4287-919e-fd9a760bc4bc-kube-api-access-smzl6\") pod \"c118d9fb-524e-4287-919e-fd9a760bc4bc\" (UID: \"c118d9fb-524e-4287-919e-fd9a760bc4bc\") " Oct 31 04:00:22.003201 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.001083 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5jrg\" (UniqueName: \"kubernetes.io/projected/ec1cd5ae-d667-484f-b302-3680f74423d9-kube-api-access-z5jrg\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:22.025072 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.025034 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c118d9fb-524e-4287-919e-fd9a760bc4bc-kube-api-access-smzl6" pod "c118d9fb-524e-4287-919e-fd9a760bc4bc" (UID: "c118d9fb-524e-4287-919e-fd9a760bc4bc"). InnerVolumeSpecName "kube-api-access-smzl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:22.043046 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.042985 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5"] Oct 31 04:00:22.047217 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.047111 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29acde0-cc51-41a1-91c4-7522b9c4978c-kube-api-access-rbtrf" pod "f29acde0-cc51-41a1-91c4-7522b9c4978c" (UID: "f29acde0-cc51-41a1-91c4-7522b9c4978c"). InnerVolumeSpecName "kube-api-access-rbtrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:22.053388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.053355 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a64fd2-29890320-vxkr5"] Oct 31 04:00:22.101840 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.101794 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6340a3dd-2df2-4aaf-9a91-1a64798bed73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6340a3dd-2df2-4aaf-9a91-1a64798bed73-maas-api-service-ca\") pod \"6340a3dd-2df2-4aaf-9a91-1a64798bed73\" (UID: \"6340a3dd-2df2-4aaf-9a91-1a64798bed73\") " Oct 31 04:00:22.103012 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.102879 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6340a3dd-2df2-4aaf-9a91-1a64798bed73-maas-api-service-ca" pod "6340a3dd-2df2-4aaf-9a91-1a64798bed73" (UID: "6340a3dd-2df2-4aaf-9a91-1a64798bed73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:22.103194 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.103016 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c118d9fb-524e-4287-919e-fd9a760bc4bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c118d9fb-524e-4287-919e-fd9a760bc4bc-maas-api-service-ca\") pod \"c118d9fb-524e-4287-919e-fd9a760bc4bc\" (UID: \"c118d9fb-524e-4287-919e-fd9a760bc4bc\") " Oct 31 04:00:22.103194 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.103079 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6340a3dd-2df2-4aaf-9a91-1a64798bed73-kube-api-access-7trbd\" (UniqueName: \"kubernetes.io/projected/6340a3dd-2df2-4aaf-9a91-1a64798bed73-kube-api-access-7trbd\") pod \"6340a3dd-2df2-4aaf-9a91-1a64798bed73\" (UID: \"6340a3dd-2df2-4aaf-9a91-1a64798bed73\") " Oct 31 04:00:22.103194 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.103161 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29acde0-cc51-41a1-91c4-7522b9c4978c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29acde0-cc51-41a1-91c4-7522b9c4978c-maas-api-service-ca\") pod \"f29acde0-cc51-41a1-91c4-7522b9c4978c\" (UID: \"f29acde0-cc51-41a1-91c4-7522b9c4978c\") " Oct 31 04:00:22.103519 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.103496 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29acde0-cc51-41a1-91c4-7522b9c4978c-maas-api-service-ca" pod "f29acde0-cc51-41a1-91c4-7522b9c4978c" (UID: "f29acde0-cc51-41a1-91c4-7522b9c4978c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:22.103615 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.103496 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c118d9fb-524e-4287-919e-fd9a760bc4bc-maas-api-service-ca" pod "c118d9fb-524e-4287-919e-fd9a760bc4bc" (UID: "c118d9fb-524e-4287-919e-fd9a760bc4bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:22.104563 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.104533 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smzl6\" (UniqueName: \"kubernetes.io/projected/c118d9fb-524e-4287-919e-fd9a760bc4bc-kube-api-access-smzl6\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:22.104664 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.104569 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6340a3dd-2df2-4aaf-9a91-1a64798bed73-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:22.104664 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.104586 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c118d9fb-524e-4287-919e-fd9a760bc4bc-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:22.104664 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.104601 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29acde0-cc51-41a1-91c4-7522b9c4978c-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:22.104664 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.104615 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbtrf\" (UniqueName: \"kubernetes.io/projected/f29acde0-cc51-41a1-91c4-7522b9c4978c-kube-api-access-rbtrf\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:22.106314 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.106286 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6340a3dd-2df2-4aaf-9a91-1a64798bed73-kube-api-access-7trbd" pod "6340a3dd-2df2-4aaf-9a91-1a64798bed73" (UID: "6340a3dd-2df2-4aaf-9a91-1a64798bed73"). InnerVolumeSpecName "kube-api-access-7trbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:22.198381 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.198347 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" Oct 31 04:00:22.205968 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.205937 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7trbd\" (UniqueName: \"kubernetes.io/projected/6340a3dd-2df2-4aaf-9a91-1a64798bed73-kube-api-access-7trbd\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:22.307095 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.306996 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c25626-c789-442c-91f4-7d453fc1e26d-kube-api-access-tq9hp\" (UniqueName: \"kubernetes.io/projected/87c25626-c789-442c-91f4-7d453fc1e26d-kube-api-access-tq9hp\") pod \"87c25626-c789-442c-91f4-7d453fc1e26d\" (UID: \"87c25626-c789-442c-91f4-7d453fc1e26d\") " Oct 31 04:00:22.307269 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.307249 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c25626-c789-442c-91f4-7d453fc1e26d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87c25626-c789-442c-91f4-7d453fc1e26d-maas-api-service-ca\") pod \"87c25626-c789-442c-91f4-7d453fc1e26d\" (UID: \"87c25626-c789-442c-91f4-7d453fc1e26d\") " Oct 31 04:00:22.307620 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.307590 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c25626-c789-442c-91f4-7d453fc1e26d-maas-api-service-ca" pod "87c25626-c789-442c-91f4-7d453fc1e26d" (UID: "87c25626-c789-442c-91f4-7d453fc1e26d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:22.307969 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.307949 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87c25626-c789-442c-91f4-7d453fc1e26d-maas-api-service-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:22.309234 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.309212 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c25626-c789-442c-91f4-7d453fc1e26d-kube-api-access-tq9hp" pod "87c25626-c789-442c-91f4-7d453fc1e26d" (UID: "87c25626-c789-442c-91f4-7d453fc1e26d"). InnerVolumeSpecName "kube-api-access-tq9hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:22.410104 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.410067 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq9hp\" (UniqueName: \"kubernetes.io/projected/87c25626-c789-442c-91f4-7d453fc1e26d-kube-api-access-tq9hp\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:22.716989 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.716956 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" Oct 31 04:00:22.717562 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.717000 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv" event={"ID":"ec1cd5ae-d667-484f-b302-3680f74423d9","Type":"ContainerDied","Data":"9ef1a8dea9d3f31368654abebbc547f0a7bfd7875e48a931f57f269bb027af11"} Oct 31 04:00:22.717562 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.717068 4426 scope.go:122] "RemoveContainer" containerID="62e3ab77972b889eda0e53eb6ffd495820407e113df38d43d5189ff42e99f93c" Oct 31 04:00:22.719031 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.719005 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" Oct 31 04:00:22.719219 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.719037 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt" event={"ID":"f29acde0-cc51-41a1-91c4-7522b9c4978c","Type":"ContainerDied","Data":"954e36bb81524ac590b2c25598ed245c819b4fa9819ea5b031f0b712641f4ddb"} Oct 31 04:00:22.720948 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.720927 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" Oct 31 04:00:22.721057 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.720998 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg" event={"ID":"6340a3dd-2df2-4aaf-9a91-1a64798bed73","Type":"ContainerDied","Data":"75f66d3803538b860b400353500d333b503e2bfc852e8ca97b0f963bdb908689"} Oct 31 04:00:22.725649 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.725625 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" event={"ID":"c118d9fb-524e-4287-919e-fd9a760bc4bc","Type":"ContainerDied","Data":"7f5b3cb1e738f90a1d9c6f3015602d31eff3dde72e3de6d561fae09dd28a3f06"} Oct 31 04:00:22.725776 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.725700 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc" Oct 31 04:00:22.729451 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.729422 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" event={"ID":"87c25626-c789-442c-91f4-7d453fc1e26d","Type":"ContainerDied","Data":"2313e64d0e691687fcd60197a256e876b47ff1e81d72029507a4e78214b4164f"} Oct 31 04:00:22.729563 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.729460 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2" Oct 31 04:00:22.733664 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.733643 4426 scope.go:122] "RemoveContainer" containerID="309799667fdccdd90bdfbe042eb3945e29089420044ed7c1d65fbf82584c12ab" Oct 31 04:00:22.748340 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.748308 4426 scope.go:122] "RemoveContainer" containerID="75515b7e9ea8bea2dbcdd320f98cafb0618ee7b1449d7a6b57e1d737a8132a8d" Oct 31 04:00:22.760238 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.760210 4426 scope.go:122] "RemoveContainer" containerID="9ecada6226f1ba1726ca26f9754842edfdd3945d8668caaabf77e2413c404f7a" Oct 31 04:00:22.771901 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.771880 4426 scope.go:122] "RemoveContainer" containerID="fecd3126104a80d04a96948ca6b509d5be4b91001df5fc2b99ca89859ececcec" Oct 31 04:00:22.882382 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.882337 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2"] Oct 31 04:00:22.888530 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.888496 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e5b8f9-29890320-8ncg2"] Oct 31 04:00:22.996298 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:22.996199 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv"] Oct 31 04:00:23.003789 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.003757 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68aa3e-29890320-j5gzv"] Oct 31 04:00:23.105847 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.105811 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc"] Oct 31 04:00:23.118916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.118854 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9hcxc"] Oct 31 04:00:23.191585 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.191544 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2ac1b0-6998-4daa-8f90-df002347e7b5" path="/var/lib/kubelet/pods/2e2ac1b0-6998-4daa-8f90-df002347e7b5/volumes" Oct 31 04:00:23.193955 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.192708 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" path="/var/lib/kubelet/pods/87c25626-c789-442c-91f4-7d453fc1e26d/volumes" Oct 31 04:00:23.195149 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.195112 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" path="/var/lib/kubelet/pods/a67baf37-cce6-4ce0-ae7a-f23571fd3ea4/volumes" Oct 31 04:00:23.197005 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.195901 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" path="/var/lib/kubelet/pods/c118d9fb-524e-4287-919e-fd9a760bc4bc/volumes" Oct 31 04:00:23.199396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.199375 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" path="/var/lib/kubelet/pods/ec1cd5ae-d667-484f-b302-3680f74423d9/volumes" Oct 31 04:00:23.200286 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.200266 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" path="/var/lib/kubelet/pods/f779ce49-773a-4bc7-87db-be2271baa5d0/volumes" Oct 31 04:00:23.271313 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.271220 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt"] Oct 31 04:00:23.293212 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.293147 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-938507-29890320-hzwjt"] Oct 31 04:00:23.355297 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.355260 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg"] Oct 31 04:00:23.361033 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:23.361001 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-537e34-29890320-zc5bg"] Oct 31 04:00:24.441709 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:24.441675 4426 scope.go:122] "RemoveContainer" containerID="d253d9e5fdc81611fe1d6e62302f522e4dcf0ae46c8d23deb29a574b457d6781" Oct 31 04:00:24.504148 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:24.495694 4426 scope.go:122] "RemoveContainer" containerID="8bdc2507ebc7337ba2277fab99f60647848a782361454a3458681df6e0dfe6bb" Oct 31 04:00:24.642403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:24.642353 4426 scope.go:122] "RemoveContainer" containerID="d2c2c3a01f5f24feb886403a79703797d995fb3637de8e229e683cd08eb6cb8c" Oct 31 04:00:24.658841 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:24.658810 4426 scope.go:122] "RemoveContainer" containerID="19398a9478ebf27c2ad306ba152231fb517747b0cbea6957c38c0b80d6be736d" Oct 31 04:00:24.692664 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:24.692634 4426 scope.go:122] "RemoveContainer" containerID="0ec6553fb396ab02d2c6994eb791bfa78af4af77d2b98ab9724a0636a25c1bab" Oct 31 04:00:24.725731 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:24.725705 4426 scope.go:122] "RemoveContainer" containerID="82dfae5a11a1a0ef54689eece6e386dd5f5f6b440af2b24c7db2a6e2fa587137" Oct 31 04:00:24.746168 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:24.746014 4426 scope.go:122] "RemoveContainer" containerID="6f5f0f148f411b269dff5e1d9b9bd22426cb72ce822eefca5c760b8e0ee9ffe9" Oct 31 04:00:24.770181 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:24.770000 4426 scope.go:122] "RemoveContainer" containerID="93dd93270a423cc9e5e3354c062a75e2bfd7b2456c9bd9470d19b96d2a9e8c96" Oct 31 04:00:24.788054 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:24.788007 4426 scope.go:122] "RemoveContainer" containerID="86eebec09fe3eaf108afb856ecc389a8d1011b875c4674fa38876ecff6402557" Oct 31 04:00:24.815984 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:24.815956 4426 scope.go:122] "RemoveContainer" containerID="e7bdcd78cf46e25ad06a07a842b7afdc4950165bde83c8ebc5dcd8a070a44d84" Oct 31 04:00:24.830282 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:24.830256 4426 scope.go:122] "RemoveContainer" containerID="f623977886c9c6fbaf7f3f52346c2c602745a65e02e2517a98aaa6962d56d07e" Oct 31 04:00:25.139264 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:25.139229 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" path="/var/lib/kubelet/pods/6340a3dd-2df2-4aaf-9a91-1a64798bed73/volumes" Oct 31 04:00:25.139806 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:25.139790 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" path="/var/lib/kubelet/pods/f29acde0-cc51-41a1-91c4-7522b9c4978c/volumes" Oct 31 04:00:27.425144 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:27.419385 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v"] Oct 31 04:00:27.425144 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:27.419616 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" podUID="4614978e-00e9-4f01-9589-af551a28666f" containerName="istio-proxy" containerID="cri-o://2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5" gracePeriod=30 Oct 31 04:00:27.463975 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:27.463935 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q"] Oct 31 04:00:27.464258 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:27.464229 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" containerID="cri-o://a2197c707bcf04689fb4299b64f4e518604d961858677c80025d167236773126" gracePeriod=30 Oct 31 04:00:27.469728 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:27.469698 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb"] Oct 31 04:00:27.469956 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:27.469905 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" containerID="cri-o://ae6253eca5fba12a780917418d266d4d7ce250cb29e92a1b1cab9d384b31f789" gracePeriod=30 Oct 31 04:00:27.510639 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:27.510600 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr"] Oct 31 04:00:27.510834 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:27.510814 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" containerID="cri-o://11faa1f8d8a95f2ddbc05de14401bb6a0bdd07fb98960fd7f9819c2ad3198d8b" gracePeriod=30 Oct 31 04:00:29.915461 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:29.915413 4426 patch_prober.go:29] interesting pod/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:00:29.915930 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:29.915490 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" podUID="4614978e-00e9-4f01-9589-af551a28666f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:00:32.901865 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.901787 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 04:00:32.902573 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.901887 4426 generic.go:356] "Generic (PLEG): container finished" podID="4614978e-00e9-4f01-9589-af551a28666f" containerID="2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5" exitCode=0 Oct 31 04:00:32.902573 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.902369 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" event={"ID":"4614978e-00e9-4f01-9589-af551a28666f","Type":"ContainerDied","Data":"2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5"} Oct 31 04:00:32.902573 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.902429 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" event={"ID":"4614978e-00e9-4f01-9589-af551a28666f","Type":"ContainerDied","Data":"71b26e7fbed1f1fc9435d3b2cd06e0d69ad726d72f617535fef86428699c47e9"} Oct 31 04:00:32.902573 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.902461 4426 scope.go:122] "RemoveContainer" containerID="2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5" Oct 31 04:00:32.903800 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:32.903747 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:00:32.906396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.906347 4426 generic.go:356] "Generic (PLEG): container finished" podID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerID="11faa1f8d8a95f2ddbc05de14401bb6a0bdd07fb98960fd7f9819c2ad3198d8b" exitCode=0 Oct 31 04:00:32.906522 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.906487 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" event={"ID":"4ebd34aa-f849-416e-9a2a-b895527619a8","Type":"ContainerDied","Data":"11faa1f8d8a95f2ddbc05de14401bb6a0bdd07fb98960fd7f9819c2ad3198d8b"} Oct 31 04:00:32.908961 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.908821 4426 generic.go:356] "Generic (PLEG): container finished" podID="43841623-83a4-4257-a478-a1665c421791" containerID="a2197c707bcf04689fb4299b64f4e518604d961858677c80025d167236773126" exitCode=0 Oct 31 04:00:32.908961 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.908885 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" event={"ID":"43841623-83a4-4257-a478-a1665c421791","Type":"ContainerDied","Data":"a2197c707bcf04689fb4299b64f4e518604d961858677c80025d167236773126"} Oct 31 04:00:32.911913 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.911626 4426 generic.go:356] "Generic (PLEG): container finished" podID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerID="ae6253eca5fba12a780917418d266d4d7ce250cb29e92a1b1cab9d384b31f789" exitCode=0 Oct 31 04:00:32.911913 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.911847 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" event={"ID":"e43b072d-0ec4-4372-9b18-3b7c94f6902f","Type":"ContainerDied","Data":"ae6253eca5fba12a780917418d266d4d7ce250cb29e92a1b1cab9d384b31f789"} Oct 31 04:00:32.924300 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.924243 4426 scope.go:122] "RemoveContainer" containerID="2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5" Oct 31 04:00:32.924982 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:00:32.924936 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5\": container with ID starting with 2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5 not found: ID does not exist" containerID="2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5" Oct 31 04:00:32.925187 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.925106 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5"} err="failed to get container status \"2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5\": rpc error: code = NotFound desc = could not find container \"2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5\": container with ID starting with 2dbeff9268646a3d36e72b6dfa3947ec43e25fbf8ef8ff0fdd7ff5eda5c1c5b5 not found: ID does not exist" Oct 31 04:00:32.969813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.966064 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-kube-api-access-cvpss\" (UniqueName: \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-kube-api-access-cvpss\") pod \"4614978e-00e9-4f01-9589-af551a28666f\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " Oct 31 04:00:32.969813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.966143 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4614978e-00e9-4f01-9589-af551a28666f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4614978e-00e9-4f01-9589-af551a28666f-istio-podinfo\") pod \"4614978e-00e9-4f01-9589-af551a28666f\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " Oct 31 04:00:32.969813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.966212 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-data\") pod \"4614978e-00e9-4f01-9589-af551a28666f\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " Oct 31 04:00:32.969813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.966242 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-istio-token\" (UniqueName: \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-istio-token\") pod \"4614978e-00e9-4f01-9589-af551a28666f\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " Oct 31 04:00:32.969813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.966276 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-certs\") pod \"4614978e-00e9-4f01-9589-af551a28666f\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " Oct 31 04:00:32.969813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.966330 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-socket\") pod \"4614978e-00e9-4f01-9589-af551a28666f\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " Oct 31 04:00:32.969813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.966386 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-envoy\") pod \"4614978e-00e9-4f01-9589-af551a28666f\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " Oct 31 04:00:32.969813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.966513 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-credential-socket\") pod \"4614978e-00e9-4f01-9589-af551a28666f\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " Oct 31 04:00:32.969813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.966572 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4614978e-00e9-4f01-9589-af551a28666f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4614978e-00e9-4f01-9589-af551a28666f-istiod-ca-cert\") pod \"4614978e-00e9-4f01-9589-af551a28666f\" (UID: \"4614978e-00e9-4f01-9589-af551a28666f\") " Oct 31 04:00:32.969813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.967660 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4614978e-00e9-4f01-9589-af551a28666f-istiod-ca-cert" pod "4614978e-00e9-4f01-9589-af551a28666f" (UID: "4614978e-00e9-4f01-9589-af551a28666f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:32.969813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.969672 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-certs" pod "4614978e-00e9-4f01-9589-af551a28666f" (UID: "4614978e-00e9-4f01-9589-af551a28666f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:32.971085 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.970584 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-kube-api-access-cvpss" pod "4614978e-00e9-4f01-9589-af551a28666f" (UID: "4614978e-00e9-4f01-9589-af551a28666f"). InnerVolumeSpecName "kube-api-access-cvpss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:32.971085 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.970804 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-credential-socket" pod "4614978e-00e9-4f01-9589-af551a28666f" (UID: "4614978e-00e9-4f01-9589-af551a28666f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:32.971085 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.970820 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-socket" pod "4614978e-00e9-4f01-9589-af551a28666f" (UID: "4614978e-00e9-4f01-9589-af551a28666f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:32.985311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.982977 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-data" pod "4614978e-00e9-4f01-9589-af551a28666f" (UID: "4614978e-00e9-4f01-9589-af551a28666f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:32.985311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.985236 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4614978e-00e9-4f01-9589-af551a28666f-istio-podinfo" pod "4614978e-00e9-4f01-9589-af551a28666f" (UID: "4614978e-00e9-4f01-9589-af551a28666f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:00:32.987177 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.987106 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-istio-token" pod "4614978e-00e9-4f01-9589-af551a28666f" (UID: "4614978e-00e9-4f01-9589-af551a28666f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:32.987982 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.987905 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 04:00:32.989156 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:32.989109 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-envoy" pod "4614978e-00e9-4f01-9589-af551a28666f" (UID: "4614978e-00e9-4f01-9589-af551a28666f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.057865 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.057766 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9"] Oct 31 04:00:33.065987 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.063658 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92ac050-bnkc9"] Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.068525 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebd34aa-f849-416e-9a2a-b895527619a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebd34aa-f849-416e-9a2a-b895527619a8-istiod-ca-cert\") pod \"4ebd34aa-f849-416e-9a2a-b895527619a8\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.068604 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-credential-socket\") pod \"4ebd34aa-f849-416e-9a2a-b895527619a8\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.068634 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-envoy\") pod \"4ebd34aa-f849-416e-9a2a-b895527619a8\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.068697 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-certs\") pod \"4ebd34aa-f849-416e-9a2a-b895527619a8\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.068777 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-data\") pod \"4ebd34aa-f849-416e-9a2a-b895527619a8\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.068891 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-token\" (UniqueName: \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-token\") pod \"4ebd34aa-f849-416e-9a2a-b895527619a8\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.069048 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-kube-api-access-9vsrg\" (UniqueName: \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-kube-api-access-9vsrg\") pod \"4ebd34aa-f849-416e-9a2a-b895527619a8\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.069184 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-socket\") pod \"4ebd34aa-f849-416e-9a2a-b895527619a8\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.069268 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-podinfo\") pod \"4ebd34aa-f849-416e-9a2a-b895527619a8\" (UID: \"4ebd34aa-f849-416e-9a2a-b895527619a8\") " Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.070705 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvpss\" (UniqueName: \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-kube-api-access-cvpss\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.070734 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4614978e-00e9-4f01-9589-af551a28666f-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.070750 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.070763 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4614978e-00e9-4f01-9589-af551a28666f-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.070780 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.070796 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.070810 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.070827 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4614978e-00e9-4f01-9589-af551a28666f-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.070842 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4614978e-00e9-4f01-9589-af551a28666f-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.071379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.070922 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-certs" pod "4ebd34aa-f849-416e-9a2a-b895527619a8" (UID: "4ebd34aa-f849-416e-9a2a-b895527619a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.073010 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.072026 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebd34aa-f849-416e-9a2a-b895527619a8-istiod-ca-cert" pod "4ebd34aa-f849-416e-9a2a-b895527619a8" (UID: "4ebd34aa-f849-416e-9a2a-b895527619a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:33.073010 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.072168 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-credential-socket" pod "4ebd34aa-f849-416e-9a2a-b895527619a8" (UID: "4ebd34aa-f849-416e-9a2a-b895527619a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.073010 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.072371 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-socket" pod "4ebd34aa-f849-416e-9a2a-b895527619a8" (UID: "4ebd34aa-f849-416e-9a2a-b895527619a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.073010 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.072648 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-data" pod "4ebd34aa-f849-416e-9a2a-b895527619a8" (UID: "4ebd34aa-f849-416e-9a2a-b895527619a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.075592 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.075045 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 04:00:33.079203 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.078075 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-envoy" pod "4ebd34aa-f849-416e-9a2a-b895527619a8" (UID: "4ebd34aa-f849-416e-9a2a-b895527619a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.079203 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.078206 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-podinfo" pod "4ebd34aa-f849-416e-9a2a-b895527619a8" (UID: "4ebd34aa-f849-416e-9a2a-b895527619a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:00:33.080208 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.079519 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 04:00:33.080826 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.080803 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-token" pod "4ebd34aa-f849-416e-9a2a-b895527619a8" (UID: "4ebd34aa-f849-416e-9a2a-b895527619a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:33.082305 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.082275 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-kube-api-access-9vsrg" pod "4ebd34aa-f849-416e-9a2a-b895527619a8" (UID: "4ebd34aa-f849-416e-9a2a-b895527619a8"). InnerVolumeSpecName "kube-api-access-9vsrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:33.135250 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.134325 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b23084-662a-40f2-a816-e752ef0cfc16" path="/var/lib/kubelet/pods/48b23084-662a-40f2-a816-e752ef0cfc16/volumes" Oct 31 04:00:33.172238 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172203 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-envoy\") pod \"43841623-83a4-4257-a478-a1665c421791\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " Oct 31 04:00:33.172403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172284 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43841623-83a4-4257-a478-a1665c421791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43841623-83a4-4257-a478-a1665c421791-istiod-ca-cert\") pod \"43841623-83a4-4257-a478-a1665c421791\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " Oct 31 04:00:33.172403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172310 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istiod-ca-cert\") pod \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " Oct 31 04:00:33.172403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172334 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-podinfo\") pod \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " Oct 31 04:00:33.172403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172360 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-data\") pod \"43841623-83a4-4257-a478-a1665c421791\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " Oct 31 04:00:33.172403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172387 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-data\") pod \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " Oct 31 04:00:33.172705 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172441 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-certs\") pod \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " Oct 31 04:00:33.172705 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172473 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-istio-token\" (UniqueName: \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-istio-token\") pod \"43841623-83a4-4257-a478-a1665c421791\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " Oct 31 04:00:33.172705 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172515 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-kube-api-access-jlwxw\" (UniqueName: \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-kube-api-access-jlwxw\") pod \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " Oct 31 04:00:33.172705 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172602 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-socket\") pod \"43841623-83a4-4257-a478-a1665c421791\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " Oct 31 04:00:33.172705 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172640 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-envoy\") pod \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " Oct 31 04:00:33.172705 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172683 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-kube-api-access-fzbnq\" (UniqueName: \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-kube-api-access-fzbnq\") pod \"43841623-83a4-4257-a478-a1665c421791\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " Oct 31 04:00:33.173034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172719 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-data" pod "43841623-83a4-4257-a478-a1665c421791" (UID: "43841623-83a4-4257-a478-a1665c421791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.173034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172743 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43841623-83a4-4257-a478-a1665c421791-istiod-ca-cert" pod "43841623-83a4-4257-a478-a1665c421791" (UID: "43841623-83a4-4257-a478-a1665c421791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:33.173034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172864 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-certs" pod "e43b072d-0ec4-4372-9b18-3b7c94f6902f" (UID: "e43b072d-0ec4-4372-9b18-3b7c94f6902f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.173034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172741 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istiod-ca-cert" pod "e43b072d-0ec4-4372-9b18-3b7c94f6902f" (UID: "e43b072d-0ec4-4372-9b18-3b7c94f6902f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:33.173034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172749 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-token\" (UniqueName: \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-token\") pod \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " Oct 31 04:00:33.173034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.172986 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-socket" pod "43841623-83a4-4257-a478-a1665c421791" (UID: "43841623-83a4-4257-a478-a1665c421791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.173034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.173025 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-credential-socket\") pod \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " Oct 31 04:00:33.173703 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.173056 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-certs\") pod \"43841623-83a4-4257-a478-a1665c421791\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " Oct 31 04:00:33.173703 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.173093 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-credential-socket\") pod \"43841623-83a4-4257-a478-a1665c421791\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " Oct 31 04:00:33.173703 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.173094 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-data" pod "e43b072d-0ec4-4372-9b18-3b7c94f6902f" (UID: "e43b072d-0ec4-4372-9b18-3b7c94f6902f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.173703 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.173246 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43841623-83a4-4257-a478-a1665c421791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43841623-83a4-4257-a478-a1665c421791-istio-podinfo\") pod \"43841623-83a4-4257-a478-a1665c421791\" (UID: \"43841623-83a4-4257-a478-a1665c421791\") " Oct 31 04:00:33.173703 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.173284 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-socket\") pod \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\" (UID: \"e43b072d-0ec4-4372-9b18-3b7c94f6902f\") " Oct 31 04:00:33.174159 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174121 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174279 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174165 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174279 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174181 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vsrg\" (UniqueName: \"kubernetes.io/projected/4ebd34aa-f849-416e-9a2a-b895527619a8-kube-api-access-9vsrg\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174279 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174197 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43841623-83a4-4257-a478-a1665c421791-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174279 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174211 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174279 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174225 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174279 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174243 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174279 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174258 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174279 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174271 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174587 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174285 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174587 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174299 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebd34aa-f849-416e-9a2a-b895527619a8-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174587 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174314 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174587 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174328 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174587 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174342 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.174587 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.174356 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebd34aa-f849-416e-9a2a-b895527619a8-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.175189 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.175167 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-credential-socket" pod "e43b072d-0ec4-4372-9b18-3b7c94f6902f" (UID: "e43b072d-0ec4-4372-9b18-3b7c94f6902f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.175326 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.175313 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-certs" pod "43841623-83a4-4257-a478-a1665c421791" (UID: "43841623-83a4-4257-a478-a1665c421791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.175444 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.175431 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-credential-socket" pod "43841623-83a4-4257-a478-a1665c421791" (UID: "43841623-83a4-4257-a478-a1665c421791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.178215 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.178188 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-socket" pod "e43b072d-0ec4-4372-9b18-3b7c94f6902f" (UID: "e43b072d-0ec4-4372-9b18-3b7c94f6902f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.180028 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.179993 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-kube-api-access-fzbnq" pod "43841623-83a4-4257-a478-a1665c421791" (UID: "43841623-83a4-4257-a478-a1665c421791"). InnerVolumeSpecName "kube-api-access-fzbnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:33.193809 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.193761 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-envoy" pod "43841623-83a4-4257-a478-a1665c421791" (UID: "43841623-83a4-4257-a478-a1665c421791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.193809 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.193774 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-podinfo" pod "e43b072d-0ec4-4372-9b18-3b7c94f6902f" (UID: "e43b072d-0ec4-4372-9b18-3b7c94f6902f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:00:33.193809 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.193785 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-envoy" pod "e43b072d-0ec4-4372-9b18-3b7c94f6902f" (UID: "e43b072d-0ec4-4372-9b18-3b7c94f6902f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:33.194046 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.193830 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-kube-api-access-jlwxw" pod "e43b072d-0ec4-4372-9b18-3b7c94f6902f" (UID: "e43b072d-0ec4-4372-9b18-3b7c94f6902f"). InnerVolumeSpecName "kube-api-access-jlwxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:33.194046 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.193864 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-token" pod "e43b072d-0ec4-4372-9b18-3b7c94f6902f" (UID: "e43b072d-0ec4-4372-9b18-3b7c94f6902f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:33.194529 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.194500 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43841623-83a4-4257-a478-a1665c421791-istio-podinfo" pod "43841623-83a4-4257-a478-a1665c421791" (UID: "43841623-83a4-4257-a478-a1665c421791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:00:33.194655 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.194637 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-istio-token" pod "43841623-83a4-4257-a478-a1665c421791" (UID: "43841623-83a4-4257-a478-a1665c421791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:33.276061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276021 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.276061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276057 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.276061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276068 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.276388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276077 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43841623-83a4-4257-a478-a1665c421791-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.276388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276088 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.276388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276096 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43841623-83a4-4257-a478-a1665c421791-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.276388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276109 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.276388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276137 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.276388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276148 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlwxw\" (UniqueName: \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-kube-api-access-jlwxw\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.276388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276157 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.276388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276172 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzbnq\" (UniqueName: \"kubernetes.io/projected/43841623-83a4-4257-a478-a1665c421791-kube-api-access-fzbnq\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.276388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.276190 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43b072d-0ec4-4372-9b18-3b7c94f6902f-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:33.929928 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.929892 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v" Oct 31 04:00:33.932456 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.932423 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" event={"ID":"4ebd34aa-f849-416e-9a2a-b895527619a8","Type":"ContainerDied","Data":"737f6895affdb037444f403ccfb88f6876878df7cd75dd69c045743e45c7382c"} Oct 31 04:00:33.932684 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.932670 4426 scope.go:122] "RemoveContainer" containerID="11faa1f8d8a95f2ddbc05de14401bb6a0bdd07fb98960fd7f9819c2ad3198d8b" Oct 31 04:00:33.933399 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.933373 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr" Oct 31 04:00:33.938282 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.938011 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" event={"ID":"43841623-83a4-4257-a478-a1665c421791","Type":"ContainerDied","Data":"9debf1a82ca140d74abfaf4bc0c0f8e24f65bb7b591ff8ac429e78f953431bca"} Oct 31 04:00:33.938282 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.938120 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q" Oct 31 04:00:33.941163 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.941113 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" event={"ID":"e43b072d-0ec4-4372-9b18-3b7c94f6902f","Type":"ContainerDied","Data":"e9dae8123836a502fca22519b2c1ef44fd38e319e31ee295702a0340dc4d64dd"} Oct 31 04:00:33.941274 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.941229 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb" Oct 31 04:00:33.952782 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.952762 4426 scope.go:122] "RemoveContainer" containerID="a2197c707bcf04689fb4299b64f4e518604d961858677c80025d167236773126" Oct 31 04:00:33.976613 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:33.976578 4426 scope.go:122] "RemoveContainer" containerID="ae6253eca5fba12a780917418d266d4d7ce250cb29e92a1b1cab9d384b31f789" Oct 31 04:00:34.094295 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:34.094253 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q"] Oct 31 04:00:34.143497 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:34.143461 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-rqg2q"] Oct 31 04:00:34.209592 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:34.209511 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v"] Oct 31 04:00:34.214655 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:34.214624 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-m264v"] Oct 31 04:00:34.401184 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:34.401139 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr"] Oct 31 04:00:34.436609 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:34.436578 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-f7lfr"] Oct 31 04:00:34.495861 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:34.495772 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb"] Oct 31 04:00:34.503052 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:34.503016 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e5b8f9-openshift-default-5f9ddc7f7c-l2rhb"] Oct 31 04:00:35.134281 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:35.134248 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43841623-83a4-4257-a478-a1665c421791" path="/var/lib/kubelet/pods/43841623-83a4-4257-a478-a1665c421791/volumes" Oct 31 04:00:35.134859 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:35.134841 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4614978e-00e9-4f01-9589-af551a28666f" path="/var/lib/kubelet/pods/4614978e-00e9-4f01-9589-af551a28666f/volumes" Oct 31 04:00:35.135353 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:35.135337 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" path="/var/lib/kubelet/pods/4ebd34aa-f849-416e-9a2a-b895527619a8/volumes" Oct 31 04:00:35.136148 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:00:35.136106 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" path="/var/lib/kubelet/pods/e43b072d-0ec4-4372-9b18-3b7c94f6902f/volumes" Oct 31 04:01:02.850294 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:01:02.850247 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:01:13.533766 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:01:13.533666 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:01:25.090818 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:01:25.087623 4426 scope.go:122] "RemoveContainer" containerID="fd163446db74a6e00ef7fda1651133a377bd90dc90e026ffe1930146e7d6e8b5" Oct 31 04:01:25.104977 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:01:25.104941 4426 scope.go:122] "RemoveContainer" containerID="dd17f88e2adf0738671da7b175633934234af4f8c4603f71fd3626f1d3c83529" Oct 31 04:01:25.116628 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:01:25.116596 4426 scope.go:122] "RemoveContainer" containerID="30de8953bce89a966991c6379c93028f8aaf45dd3b80ce2b6c2ace078ff62f8c" Oct 31 04:01:25.128658 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:01:25.128638 4426 scope.go:122] "RemoveContainer" containerID="15e95a3adadcefe7bc627bd5a173fad3385c105cda5c74ac92651eaef45bb9d0" Oct 31 04:01:25.141073 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:01:25.141048 4426 scope.go:122] "RemoveContainer" containerID="85633e16c63d1f625ca1c2b8dda356b9f3a4bfeca87735f02c3d0838e82f88d8" Oct 31 04:01:25.153884 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:01:25.153857 4426 scope.go:122] "RemoveContainer" containerID="a25eaf076a3314c8651ee99cb23787ac54584b78bb137ee45e90a833d24e5a35" Oct 31 04:01:25.167669 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:01:25.167644 4426 scope.go:122] "RemoveContainer" containerID="22cceb905646fa1f049e072f5e71e78bbf59956b34019f93b0b9fa22bfa30f94" Oct 31 04:01:25.193513 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:01:25.193488 4426 scope.go:122] "RemoveContainer" containerID="48f39c9c49e317a70ec43d26feb25fed5294a750b5cd622c938ef2c1b8679f49" Oct 31 04:01:32.847586 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:01:32.847546 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:02:02.848594 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:02:02.848554 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:02:14.051537 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:14.051500 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996"] Oct 31 04:02:14.096083 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:14.096051 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ffba08-jc996"] Oct 31 04:02:15.132466 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:15.132427 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f845d43a-7e97-4986-acf6-cc3524d1b99f" path="/var/lib/kubelet/pods/f845d43a-7e97-4986-acf6-cc3524d1b99f/volumes" Oct 31 04:02:25.301452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:25.301417 4426 scope.go:122] "RemoveContainer" containerID="5ed1198979af2539017324a79eb59bf9c663b50ea4d83a79b096cbe1f54f8c2f" Oct 31 04:02:32.128369 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.128326 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf"] Oct 31 04:02:32.129198 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129180 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerName="cleanup" Oct 31 04:02:32.129251 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129200 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerName="cleanup" Oct 31 04:02:32.129251 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129216 4426 cpu_manager.go:414] "containerMap: removing container" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerName="cleanup" Oct 31 04:02:32.129251 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129221 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerName="cleanup" Oct 31 04:02:32.129251 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129232 4426 cpu_manager.go:414] "containerMap: removing container" podUID="4614978e-00e9-4f01-9589-af551a28666f" containerName="istio-proxy" Oct 31 04:02:32.129251 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129237 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4614978e-00e9-4f01-9589-af551a28666f" containerName="istio-proxy" Oct 31 04:02:32.129251 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129244 4426 cpu_manager.go:414] "containerMap: removing container" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" Oct 31 04:02:32.129251 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129248 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129259 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129264 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129270 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129275 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129284 4426 cpu_manager.go:414] "containerMap: removing container" podUID="1b647cae-10e2-44e2-8359-241944b5acfe" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129290 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b647cae-10e2-44e2-8359-241944b5acfe" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129297 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129301 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129310 4426 cpu_manager.go:414] "containerMap: removing container" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129314 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129322 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129326 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129334 4426 cpu_manager.go:414] "containerMap: removing container" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129338 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129343 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129348 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129362 4426 cpu_manager.go:414] "containerMap: removing container" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129367 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129375 4426 cpu_manager.go:414] "containerMap: removing container" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129380 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129387 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129392 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129398 4426 cpu_manager.go:414] "containerMap: removing container" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129403 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129409 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129414 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129420 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129426 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129432 4426 cpu_manager.go:414] "containerMap: removing container" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129436 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129443 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129447 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129457 4426 cpu_manager.go:414] "containerMap: removing container" podUID="1b647cae-10e2-44e2-8359-241944b5acfe" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129461 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b647cae-10e2-44e2-8359-241944b5acfe" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129467 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129472 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129478 4426 cpu_manager.go:414] "containerMap: removing container" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerName="cleanup" Oct 31 04:02:32.129472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129482 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129491 4426 cpu_manager.go:414] "containerMap: removing container" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129496 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129505 4426 cpu_manager.go:414] "containerMap: removing container" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129510 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129517 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129521 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129528 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129532 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129658 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="4614978e-00e9-4f01-9589-af551a28666f" containerName="istio-proxy" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129668 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43b072d-0ec4-4372-9b18-3b7c94f6902f" containerName="istio-proxy" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129675 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129680 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129688 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b647cae-10e2-44e2-8359-241944b5acfe" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129693 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129699 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129705 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129710 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129715 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129723 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129729 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="43841623-83a4-4257-a478-a1665c421791" containerName="istio-proxy" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129734 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebd34aa-f849-416e-9a2a-b895527619a8" containerName="istio-proxy" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129740 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129744 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129749 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129754 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129759 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129764 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1cd5ae-d667-484f-b302-3680f74423d9" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129768 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129775 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c25626-c789-442c-91f4-7d453fc1e26d" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129781 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129786 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129791 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129797 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="6340a3dd-2df2-4aaf-9a91-1a64798bed73" containerName="cleanup" Oct 31 04:02:32.130734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.129803 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerName="cleanup" Oct 31 04:02:32.135756 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.135737 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:32.157239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.157205 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-22e20532-fdnhf\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:32.157373 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.157315 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l7cn\" (UniqueName: \"kubernetes.io/projected/52ffd0b4-21d4-4728-8b14-526aebb469dc-kube-api-access-9l7cn\") pod \"maas-api-revoke-keys-e2e-mt-22e20532-fdnhf\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:32.157569 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.157556 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-22e20532-fdnhf\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:32.187818 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.187779 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf"] Oct 31 04:02:32.228339 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.227679 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:02:32.228339 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.227683 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:02:32.228339 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.227764 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:02:32.230791 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.230770 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:02:32.258557 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.258522 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-22e20532-fdnhf\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:32.258741 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.258595 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-22e20532-fdnhf\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:32.258741 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.258641 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l7cn\" (UniqueName: \"kubernetes.io/projected/52ffd0b4-21d4-4728-8b14-526aebb469dc-kube-api-access-9l7cn\") pod \"maas-api-revoke-keys-e2e-mt-22e20532-fdnhf\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:32.259419 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.259391 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-22e20532-fdnhf\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:32.261083 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.261048 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-22e20532-fdnhf\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:32.316186 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.316146 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l7cn\" (UniqueName: \"kubernetes.io/projected/52ffd0b4-21d4-4728-8b14-526aebb469dc-kube-api-access-9l7cn\") pod \"maas-api-revoke-keys-e2e-mt-22e20532-fdnhf\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:32.450937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.450895 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:32.848118 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:02:32.848078 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:02:32.855931 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:32.855908 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf"] Oct 31 04:02:32.858974 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:02:32.858947 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ffd0b4_21d4_4728_8b14_526aebb469dc.slice/crio-a022c5f8167acaf084c783dabf42b8cc166a31262cccafa5cf4c0ad932493d23 WatchSource:0}: Error finding container a022c5f8167acaf084c783dabf42b8cc166a31262cccafa5cf4c0ad932493d23: Status 404 returned error can't find the container with id a022c5f8167acaf084c783dabf42b8cc166a31262cccafa5cf4c0ad932493d23 Oct 31 04:02:33.606403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:33.606311 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" event={"ID":"52ffd0b4-21d4-4728-8b14-526aebb469dc","Type":"ContainerStarted","Data":"49cdca9556969fa5823846ede5b6ee5e05136791787cd896f414cdc6d3c11c26"} Oct 31 04:02:33.606403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:33.606381 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" event={"ID":"52ffd0b4-21d4-4728-8b14-526aebb469dc","Type":"ContainerStarted","Data":"a022c5f8167acaf084c783dabf42b8cc166a31262cccafa5cf4c0ad932493d23"} Oct 31 04:02:33.716497 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:33.716439 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" podStartSLOduration=1.7164242490000001 podStartE2EDuration="1.716424249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:33.715436134 +0000 UTC m=+2050.716061362" watchObservedRunningTime="2026-10-31 04:02:33.716424249 +0000 UTC m=+2050.717049484" Oct 31 04:02:34.618751 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:34.618706 4426 generic.go:356] "Generic (PLEG): container finished" podID="52ffd0b4-21d4-4728-8b14-526aebb469dc" containerID="49cdca9556969fa5823846ede5b6ee5e05136791787cd896f414cdc6d3c11c26" exitCode=0 Oct 31 04:02:34.619216 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:34.618796 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" event={"ID":"52ffd0b4-21d4-4728-8b14-526aebb469dc","Type":"ContainerDied","Data":"49cdca9556969fa5823846ede5b6ee5e05136791787cd896f414cdc6d3c11c26"} Oct 31 04:02:35.815356 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:35.815323 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:35.925831 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:35.925792 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-openshift-ca\") pod \"52ffd0b4-21d4-4728-8b14-526aebb469dc\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " Oct 31 04:02:35.926035 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:35.925872 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ffd0b4-21d4-4728-8b14-526aebb469dc-kube-api-access-9l7cn\" (UniqueName: \"kubernetes.io/projected/52ffd0b4-21d4-4728-8b14-526aebb469dc-kube-api-access-9l7cn\") pod \"52ffd0b4-21d4-4728-8b14-526aebb469dc\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " Oct 31 04:02:35.926035 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:35.926002 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-xks-ca\") pod \"52ffd0b4-21d4-4728-8b14-526aebb469dc\" (UID: \"52ffd0b4-21d4-4728-8b14-526aebb469dc\") " Oct 31 04:02:35.926300 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:35.926274 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-openshift-ca" pod "52ffd0b4-21d4-4728-8b14-526aebb469dc" (UID: "52ffd0b4-21d4-4728-8b14-526aebb469dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:02:35.926771 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:35.926746 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:35.928439 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:35.928327 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-xks-ca" pod "52ffd0b4-21d4-4728-8b14-526aebb469dc" (UID: "52ffd0b4-21d4-4728-8b14-526aebb469dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:02:35.928550 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:35.928532 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ffd0b4-21d4-4728-8b14-526aebb469dc-kube-api-access-9l7cn" pod "52ffd0b4-21d4-4728-8b14-526aebb469dc" (UID: "52ffd0b4-21d4-4728-8b14-526aebb469dc"). InnerVolumeSpecName "kube-api-access-9l7cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:02:36.029081 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:36.029041 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l7cn\" (UniqueName: \"kubernetes.io/projected/52ffd0b4-21d4-4728-8b14-526aebb469dc-kube-api-access-9l7cn\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:36.029081 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:36.029082 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ffd0b4-21d4-4728-8b14-526aebb469dc-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:36.643198 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:36.643164 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" Oct 31 04:02:36.643386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:36.643160 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf" event={"ID":"52ffd0b4-21d4-4728-8b14-526aebb469dc","Type":"ContainerDied","Data":"a022c5f8167acaf084c783dabf42b8cc166a31262cccafa5cf4c0ad932493d23"} Oct 31 04:02:36.643386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:36.643316 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a022c5f8167acaf084c783dabf42b8cc166a31262cccafa5cf4c0ad932493d23" Oct 31 04:02:42.297859 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:42.297825 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:02:45.722482 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:45.722441 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d"] Oct 31 04:02:45.825217 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:45.825167 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj"] Oct 31 04:02:50.724049 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:50.723986 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" podUID="514aed0d-15b6-4569-9139-2362d7f0ca38" containerName="payload-processing" containerID="cri-o://83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741" gracePeriod=25 Oct 31 04:02:50.826083 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:50.826013 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" podUID="06bb90ba-05f7-44c4-a917-35de0ac617dd" containerName="payload-pre-processing" containerID="cri-o://3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598" gracePeriod=25 Oct 31 04:02:51.036684 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.032924 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 04:02:51.148176 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.135011 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514aed0d-15b6-4569-9139-2362d7f0ca38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514aed0d-15b6-4569-9139-2362d7f0ca38-plugins-config-volume\") pod \"514aed0d-15b6-4569-9139-2362d7f0ca38\" (UID: \"514aed0d-15b6-4569-9139-2362d7f0ca38\") " Oct 31 04:02:51.148176 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.135106 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514aed0d-15b6-4569-9139-2362d7f0ca38-kube-api-access-zckbd\" (UniqueName: \"kubernetes.io/projected/514aed0d-15b6-4569-9139-2362d7f0ca38-kube-api-access-zckbd\") pod \"514aed0d-15b6-4569-9139-2362d7f0ca38\" (UID: \"514aed0d-15b6-4569-9139-2362d7f0ca38\") " Oct 31 04:02:51.148176 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.138317 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514aed0d-15b6-4569-9139-2362d7f0ca38-plugins-config-volume" pod "514aed0d-15b6-4569-9139-2362d7f0ca38" (UID: "514aed0d-15b6-4569-9139-2362d7f0ca38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:02:51.148176 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.138824 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514aed0d-15b6-4569-9139-2362d7f0ca38-kube-api-access-zckbd" pod "514aed0d-15b6-4569-9139-2362d7f0ca38" (UID: "514aed0d-15b6-4569-9139-2362d7f0ca38"). InnerVolumeSpecName "kube-api-access-zckbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:02:51.173806 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.173777 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 04:02:51.237033 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.236989 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bb90ba-05f7-44c4-a917-35de0ac617dd-kube-api-access-t229q\" (UniqueName: \"kubernetes.io/projected/06bb90ba-05f7-44c4-a917-35de0ac617dd-kube-api-access-t229q\") pod \"06bb90ba-05f7-44c4-a917-35de0ac617dd\" (UID: \"06bb90ba-05f7-44c4-a917-35de0ac617dd\") " Oct 31 04:02:51.237292 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.237271 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06bb90ba-05f7-44c4-a917-35de0ac617dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06bb90ba-05f7-44c4-a917-35de0ac617dd-plugins-config-volume\") pod \"06bb90ba-05f7-44c4-a917-35de0ac617dd\" (UID: \"06bb90ba-05f7-44c4-a917-35de0ac617dd\") " Oct 31 04:02:51.237737 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.237705 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bb90ba-05f7-44c4-a917-35de0ac617dd-plugins-config-volume" pod "06bb90ba-05f7-44c4-a917-35de0ac617dd" (UID: "06bb90ba-05f7-44c4-a917-35de0ac617dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:02:51.238470 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.238443 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514aed0d-15b6-4569-9139-2362d7f0ca38-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:51.238470 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.238473 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zckbd\" (UniqueName: \"kubernetes.io/projected/514aed0d-15b6-4569-9139-2362d7f0ca38-kube-api-access-zckbd\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:51.238647 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.238489 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06bb90ba-05f7-44c4-a917-35de0ac617dd-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:51.239447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.239390 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bb90ba-05f7-44c4-a917-35de0ac617dd-kube-api-access-t229q" pod "06bb90ba-05f7-44c4-a917-35de0ac617dd" (UID: "06bb90ba-05f7-44c4-a917-35de0ac617dd"). InnerVolumeSpecName "kube-api-access-t229q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:02:51.341055 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.340394 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t229q\" (UniqueName: \"kubernetes.io/projected/06bb90ba-05f7-44c4-a917-35de0ac617dd-kube-api-access-t229q\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:51.789691 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.789650 4426 generic.go:356] "Generic (PLEG): container finished" podID="514aed0d-15b6-4569-9139-2362d7f0ca38" containerID="83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741" exitCode=0 Oct 31 04:02:51.790210 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.789726 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" Oct 31 04:02:51.790210 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.789728 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" event={"ID":"514aed0d-15b6-4569-9139-2362d7f0ca38","Type":"ContainerDied","Data":"83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741"} Oct 31 04:02:51.790210 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.789824 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d" event={"ID":"514aed0d-15b6-4569-9139-2362d7f0ca38","Type":"ContainerDied","Data":"fb8762786b10508b889f3a636bac53ac06aec5332444389b8e9e28b7bd162569"} Oct 31 04:02:51.790210 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.789853 4426 scope.go:122] "RemoveContainer" containerID="83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741" Oct 31 04:02:51.792024 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.791995 4426 generic.go:356] "Generic (PLEG): container finished" podID="06bb90ba-05f7-44c4-a917-35de0ac617dd" containerID="3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598" exitCode=0 Oct 31 04:02:51.792180 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.792079 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" Oct 31 04:02:51.792180 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.792101 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" event={"ID":"06bb90ba-05f7-44c4-a917-35de0ac617dd","Type":"ContainerDied","Data":"3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598"} Oct 31 04:02:51.792341 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.792182 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj" event={"ID":"06bb90ba-05f7-44c4-a917-35de0ac617dd","Type":"ContainerDied","Data":"712b0405eb50addc184c889ee8891cf0934a5a992fbe218c22a1a947b125bdcb"} Oct 31 04:02:51.803186 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.803164 4426 scope.go:122] "RemoveContainer" containerID="83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741" Oct 31 04:02:51.803474 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:02:51.803454 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741\": container with ID starting with 83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741 not found: ID does not exist" containerID="83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741" Oct 31 04:02:51.803542 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.803482 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741"} err="failed to get container status \"83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741\": rpc error: code = NotFound desc = could not find container \"83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741\": container with ID starting with 83ad00faedcb54cbdac316722a8a3ed41be8fc50f8a98f2b6000a811aad8e741 not found: ID does not exist" Oct 31 04:02:51.803542 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.803500 4426 scope.go:122] "RemoveContainer" containerID="3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598" Oct 31 04:02:51.820071 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.820047 4426 scope.go:122] "RemoveContainer" containerID="3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598" Oct 31 04:02:51.820453 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:02:51.820429 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598\": container with ID starting with 3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598 not found: ID does not exist" containerID="3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598" Oct 31 04:02:51.820541 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.820466 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598"} err="failed to get container status \"3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598\": rpc error: code = NotFound desc = could not find container \"3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598\": container with ID starting with 3f9f248f168d8847a81f7c28f9c59c5c1de3dbcc65ee31f3c3d3b65209466598 not found: ID does not exist" Oct 31 04:02:51.971189 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.971151 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d"] Oct 31 04:02:51.992735 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:51.992597 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22e20532-6d8d97dd77-kgj7d"] Oct 31 04:02:52.073705 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:52.073603 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj"] Oct 31 04:02:52.083796 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:52.083733 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22e20532-d8c4c5cdc-ktpxj"] Oct 31 04:02:53.134372 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:53.134338 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06bb90ba-05f7-44c4-a917-35de0ac617dd" path="/var/lib/kubelet/pods/06bb90ba-05f7-44c4-a917-35de0ac617dd/volumes" Oct 31 04:02:53.134961 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:53.134944 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514aed0d-15b6-4569-9139-2362d7f0ca38" path="/var/lib/kubelet/pods/514aed0d-15b6-4569-9139-2362d7f0ca38/volumes" Oct 31 04:02:57.233189 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:57.233148 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl"] Oct 31 04:02:57.233652 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:57.233395 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerName="istio-proxy" containerID="cri-o://70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099" gracePeriod=30 Oct 31 04:02:57.258059 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:57.258018 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf"] Oct 31 04:02:57.261772 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:57.261724 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" podUID="a303237c-f769-4565-bb71-de2d23a0519b" containerName="istio-proxy" containerID="cri-o://755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226" gracePeriod=30 Oct 31 04:02:57.682814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:57.682773 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8"] Oct 31 04:02:57.683044 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:57.683015 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" containerID="cri-o://9a198853c7f27a32c63e044fdff69d060af08f0465ea99c48a1fd48a2a45c5f1" gracePeriod=30 Oct 31 04:02:58.048058 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:58.047960 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph"] Oct 31 04:02:58.052745 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:58.052712 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea6f4e33-xv6ph"] Oct 31 04:02:58.280597 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:58.280562 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs"] Oct 31 04:02:58.281065 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:58.280765 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" containerID="cri-o://122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341" gracePeriod=30 Oct 31 04:02:59.132827 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:59.132797 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ff520e-3694-4628-8840-f54fbc9ef3cd" path="/var/lib/kubelet/pods/68ff520e-3694-4628-8840-f54fbc9ef3cd/volumes" Oct 31 04:02:59.137172 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:59.137139 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:02:59.137306 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:02:59.137203 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:03:01.001877 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:01.001835 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:03:01.002458 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:01.001907 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:03:01.712721 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:01.712676 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:03:01.712904 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:01.712743 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" podUID="a303237c-f769-4565-bb71-de2d23a0519b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:03:02.633156 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.628943 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 04:03:02.706167 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.704752 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-podinfo\") pod \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " Oct 31 04:03:02.706167 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.705034 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-socket\") pod \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " Oct 31 04:03:02.706167 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.705167 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-data\") pod \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " Oct 31 04:03:02.706167 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.705207 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-certs\") pod \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " Oct 31 04:03:02.706167 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.705247 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-credential-socket\") pod \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " Oct 31 04:03:02.706167 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.705322 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-token\" (UniqueName: \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-token\") pod \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " Oct 31 04:03:02.706167 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.705365 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-kube-api-access-rmg7b\" (UniqueName: \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-kube-api-access-rmg7b\") pod \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " Oct 31 04:03:02.706167 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.705401 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istiod-ca-cert\") pod \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " Oct 31 04:03:02.706167 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.705510 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-envoy\") pod \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\" (UID: \"b790d330-2c37-4ce4-8cf3-57afa5e12d63\") " Oct 31 04:03:02.712294 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.709198 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-envoy" pod "b790d330-2c37-4ce4-8cf3-57afa5e12d63" (UID: "b790d330-2c37-4ce4-8cf3-57afa5e12d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:02.712294 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.709480 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-credential-socket" pod "b790d330-2c37-4ce4-8cf3-57afa5e12d63" (UID: "b790d330-2c37-4ce4-8cf3-57afa5e12d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:02.712294 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.709834 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istiod-ca-cert" pod "b790d330-2c37-4ce4-8cf3-57afa5e12d63" (UID: "b790d330-2c37-4ce4-8cf3-57afa5e12d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:03:02.712294 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.710437 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-socket" pod "b790d330-2c37-4ce4-8cf3-57afa5e12d63" (UID: "b790d330-2c37-4ce4-8cf3-57afa5e12d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:02.712294 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.711389 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-data" pod "b790d330-2c37-4ce4-8cf3-57afa5e12d63" (UID: "b790d330-2c37-4ce4-8cf3-57afa5e12d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:02.712674 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.712449 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-kube-api-access-rmg7b" pod "b790d330-2c37-4ce4-8cf3-57afa5e12d63" (UID: "b790d330-2c37-4ce4-8cf3-57afa5e12d63"). InnerVolumeSpecName "kube-api-access-rmg7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:02.715955 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.715072 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-token" pod "b790d330-2c37-4ce4-8cf3-57afa5e12d63" (UID: "b790d330-2c37-4ce4-8cf3-57afa5e12d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:02.715955 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.715197 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-certs" pod "b790d330-2c37-4ce4-8cf3-57afa5e12d63" (UID: "b790d330-2c37-4ce4-8cf3-57afa5e12d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:02.736902 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.736508 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 04:03:02.736902 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.736844 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-podinfo" pod "b790d330-2c37-4ce4-8cf3-57afa5e12d63" (UID: "b790d330-2c37-4ce4-8cf3-57afa5e12d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:03:02.807937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.807826 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:02.807937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.807872 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:02.807937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.807886 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:02.807937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.807900 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:02.807937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.807914 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:02.807937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.807928 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmg7b\" (UniqueName: \"kubernetes.io/projected/b790d330-2c37-4ce4-8cf3-57afa5e12d63-kube-api-access-rmg7b\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:02.808363 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.807942 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:02.808363 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.807955 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:02.808363 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.807970 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b790d330-2c37-4ce4-8cf3-57afa5e12d63-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:02.862158 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:03:02.862097 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:03:02.908670 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.908639 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-socket\") pod \"a303237c-f769-4565-bb71-de2d23a0519b\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " Oct 31 04:03:02.908942 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.908905 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-socket" pod "a303237c-f769-4565-bb71-de2d23a0519b" (UID: "a303237c-f769-4565-bb71-de2d23a0519b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:02.909048 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.908950 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a303237c-f769-4565-bb71-de2d23a0519b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a303237c-f769-4565-bb71-de2d23a0519b-istiod-ca-cert\") pod \"a303237c-f769-4565-bb71-de2d23a0519b\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " Oct 31 04:03:02.909048 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.909031 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-envoy\") pod \"a303237c-f769-4565-bb71-de2d23a0519b\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " Oct 31 04:03:02.909329 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.909138 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-certs\") pod \"a303237c-f769-4565-bb71-de2d23a0519b\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " Oct 31 04:03:02.909329 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.909214 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a303237c-f769-4565-bb71-de2d23a0519b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a303237c-f769-4565-bb71-de2d23a0519b-istio-podinfo\") pod \"a303237c-f769-4565-bb71-de2d23a0519b\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " Oct 31 04:03:02.909329 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.909245 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-data\") pod \"a303237c-f769-4565-bb71-de2d23a0519b\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " Oct 31 04:03:02.909329 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.909272 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-kube-api-access-6brhl\" (UniqueName: \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-kube-api-access-6brhl\") pod \"a303237c-f769-4565-bb71-de2d23a0519b\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " Oct 31 04:03:02.909329 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.909297 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-istio-token\" (UniqueName: \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-istio-token\") pod \"a303237c-f769-4565-bb71-de2d23a0519b\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " Oct 31 04:03:02.909721 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.909353 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-credential-socket\") pod \"a303237c-f769-4565-bb71-de2d23a0519b\" (UID: \"a303237c-f769-4565-bb71-de2d23a0519b\") " Oct 31 04:03:02.909801 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.909782 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:02.910435 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.909999 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-credential-socket" pod "a303237c-f769-4565-bb71-de2d23a0519b" (UID: "a303237c-f769-4565-bb71-de2d23a0519b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:02.910435 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.910342 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-data" pod "a303237c-f769-4565-bb71-de2d23a0519b" (UID: "a303237c-f769-4565-bb71-de2d23a0519b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:02.910919 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.910884 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a303237c-f769-4565-bb71-de2d23a0519b-istiod-ca-cert" pod "a303237c-f769-4565-bb71-de2d23a0519b" (UID: "a303237c-f769-4565-bb71-de2d23a0519b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:03:02.911223 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.911183 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-certs" pod "a303237c-f769-4565-bb71-de2d23a0519b" (UID: "a303237c-f769-4565-bb71-de2d23a0519b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:02.912744 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.912706 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-kube-api-access-6brhl" pod "a303237c-f769-4565-bb71-de2d23a0519b" (UID: "a303237c-f769-4565-bb71-de2d23a0519b"). InnerVolumeSpecName "kube-api-access-6brhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:02.913052 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.913024 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a303237c-f769-4565-bb71-de2d23a0519b-istio-podinfo" pod "a303237c-f769-4565-bb71-de2d23a0519b" (UID: "a303237c-f769-4565-bb71-de2d23a0519b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:03:02.913358 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.913334 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-envoy" pod "a303237c-f769-4565-bb71-de2d23a0519b" (UID: "a303237c-f769-4565-bb71-de2d23a0519b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:02.914309 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.914283 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-istio-token" pod "a303237c-f769-4565-bb71-de2d23a0519b" (UID: "a303237c-f769-4565-bb71-de2d23a0519b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:02.922451 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.922254 4426 generic.go:356] "Generic (PLEG): container finished" podID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerID="9a198853c7f27a32c63e044fdff69d060af08f0465ea99c48a1fd48a2a45c5f1" exitCode=0 Oct 31 04:03:02.922724 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.922699 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" event={"ID":"5ef90cb4-6f28-437e-a875-80d202b3f16c","Type":"ContainerDied","Data":"9a198853c7f27a32c63e044fdff69d060af08f0465ea99c48a1fd48a2a45c5f1"} Oct 31 04:03:02.928165 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.924580 4426 generic.go:356] "Generic (PLEG): container finished" podID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerID="70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099" exitCode=0 Oct 31 04:03:02.928165 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.924774 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" Oct 31 04:03:02.942177 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.941020 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" event={"ID":"b790d330-2c37-4ce4-8cf3-57afa5e12d63","Type":"ContainerDied","Data":"70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099"} Oct 31 04:03:02.942177 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.941112 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl" event={"ID":"b790d330-2c37-4ce4-8cf3-57afa5e12d63","Type":"ContainerDied","Data":"306bb1471117ef7994b5d9d990cbd85da373b022c7ae43ab9be1cd0697394ed2"} Oct 31 04:03:02.942177 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.941162 4426 scope.go:122] "RemoveContainer" containerID="70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099" Oct 31 04:03:02.954157 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.953657 4426 generic.go:356] "Generic (PLEG): container finished" podID="a303237c-f769-4565-bb71-de2d23a0519b" containerID="755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226" exitCode=0 Oct 31 04:03:02.954157 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.953731 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" event={"ID":"a303237c-f769-4565-bb71-de2d23a0519b","Type":"ContainerDied","Data":"755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226"} Oct 31 04:03:02.954157 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.953800 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" event={"ID":"a303237c-f769-4565-bb71-de2d23a0519b","Type":"ContainerDied","Data":"690686a732f0238f5729ed44c64698ce50abf26822a7f1390111749e46b5ab99"} Oct 31 04:03:02.954157 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:02.953887 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf" Oct 31 04:03:03.012476 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.012440 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a303237c-f769-4565-bb71-de2d23a0519b-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.012891 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.012872 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.013005 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.012996 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.013065 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.013058 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a303237c-f769-4565-bb71-de2d23a0519b-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.013200 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.013169 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.013312 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.013299 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6brhl\" (UniqueName: \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-kube-api-access-6brhl\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.013409 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.013400 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a303237c-f769-4565-bb71-de2d23a0519b-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.013638 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.013627 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a303237c-f769-4565-bb71-de2d23a0519b-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.027112 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.027093 4426 scope.go:122] "RemoveContainer" containerID="70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099" Oct 31 04:03:03.027548 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:03:03.027525 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099\": container with ID starting with 70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099 not found: ID does not exist" containerID="70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099" Oct 31 04:03:03.027661 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.027638 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099"} err="failed to get container status \"70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099\": rpc error: code = NotFound desc = could not find container \"70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099\": container with ID starting with 70b30df358907cef4e684f3481729dfc6f67e79b914a6b1d9de752de3ac0b099 not found: ID does not exist" Oct 31 04:03:03.027747 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.027738 4426 scope.go:122] "RemoveContainer" containerID="755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226" Oct 31 04:03:03.047089 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.047061 4426 scope.go:122] "RemoveContainer" containerID="755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226" Oct 31 04:03:03.047551 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:03:03.047520 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226\": container with ID starting with 755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226 not found: ID does not exist" containerID="755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226" Oct 31 04:03:03.047710 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.047562 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226"} err="failed to get container status \"755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226\": rpc error: code = NotFound desc = could not find container \"755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226\": container with ID starting with 755d1cd6fa991d841768274b48562b77e459796d783d747f136c8ce0f5570226 not found: ID does not exist" Oct 31 04:03:03.050701 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.050612 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 04:03:03.205947 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.205910 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl"] Oct 31 04:03:03.216055 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216023 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-data\") pod \"5ef90cb4-6f28-437e-a875-80d202b3f16c\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " Oct 31 04:03:03.216210 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216072 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-kube-api-access-68dg8\" (UniqueName: \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-kube-api-access-68dg8\") pod \"5ef90cb4-6f28-437e-a875-80d202b3f16c\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " Oct 31 04:03:03.216325 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216309 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-token\" (UniqueName: \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-token\") pod \"5ef90cb4-6f28-437e-a875-80d202b3f16c\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " Oct 31 04:03:03.216375 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216344 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-certs\") pod \"5ef90cb4-6f28-437e-a875-80d202b3f16c\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " Oct 31 04:03:03.216375 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216345 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-data" pod "5ef90cb4-6f28-437e-a875-80d202b3f16c" (UID: "5ef90cb4-6f28-437e-a875-80d202b3f16c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:03.216375 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216371 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-envoy\") pod \"5ef90cb4-6f28-437e-a875-80d202b3f16c\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " Oct 31 04:03:03.216516 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216429 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-podinfo\") pod \"5ef90cb4-6f28-437e-a875-80d202b3f16c\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " Oct 31 04:03:03.216516 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216476 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-credential-socket\") pod \"5ef90cb4-6f28-437e-a875-80d202b3f16c\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " Oct 31 04:03:03.216630 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216519 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-certs" pod "5ef90cb4-6f28-437e-a875-80d202b3f16c" (UID: "5ef90cb4-6f28-437e-a875-80d202b3f16c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:03.216630 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216525 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-socket\") pod \"5ef90cb4-6f28-437e-a875-80d202b3f16c\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " Oct 31 04:03:03.216740 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216678 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef90cb4-6f28-437e-a875-80d202b3f16c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef90cb4-6f28-437e-a875-80d202b3f16c-istiod-ca-cert\") pod \"5ef90cb4-6f28-437e-a875-80d202b3f16c\" (UID: \"5ef90cb4-6f28-437e-a875-80d202b3f16c\") " Oct 31 04:03:03.216800 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216766 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-socket" pod "5ef90cb4-6f28-437e-a875-80d202b3f16c" (UID: "5ef90cb4-6f28-437e-a875-80d202b3f16c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:03.216800 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.216780 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-credential-socket" pod "5ef90cb4-6f28-437e-a875-80d202b3f16c" (UID: "5ef90cb4-6f28-437e-a875-80d202b3f16c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:03.217070 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.217044 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef90cb4-6f28-437e-a875-80d202b3f16c-istiod-ca-cert" pod "5ef90cb4-6f28-437e-a875-80d202b3f16c" (UID: "5ef90cb4-6f28-437e-a875-80d202b3f16c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:03:03.217443 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.217424 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.217517 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.217447 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.217517 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.217461 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.217517 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.217474 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef90cb4-6f28-437e-a875-80d202b3f16c-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.217517 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.217491 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.218511 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.218489 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-kube-api-access-68dg8" pod "5ef90cb4-6f28-437e-a875-80d202b3f16c" (UID: "5ef90cb4-6f28-437e-a875-80d202b3f16c"). InnerVolumeSpecName "kube-api-access-68dg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:03.218661 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.218642 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-envoy" pod "5ef90cb4-6f28-437e-a875-80d202b3f16c" (UID: "5ef90cb4-6f28-437e-a875-80d202b3f16c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:03.218934 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.218897 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-podinfo" pod "5ef90cb4-6f28-437e-a875-80d202b3f16c" (UID: "5ef90cb4-6f28-437e-a875-80d202b3f16c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:03:03.218986 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.218899 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-token" pod "5ef90cb4-6f28-437e-a875-80d202b3f16c" (UID: "5ef90cb4-6f28-437e-a875-80d202b3f16c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:03.225919 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.225889 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-lcqgl"] Oct 31 04:03:03.318876 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.318844 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.318876 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.318875 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.319051 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.318885 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef90cb4-6f28-437e-a875-80d202b3f16c-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.319051 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.318895 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68dg8\" (UniqueName: \"kubernetes.io/projected/5ef90cb4-6f28-437e-a875-80d202b3f16c-kube-api-access-68dg8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.328341 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.328307 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf"] Oct 31 04:03:03.333167 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.333115 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d6zpf"] Oct 31 04:03:03.653215 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.653179 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 04:03:03.828068 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.827965 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-socket\") pod \"61524595-92d6-418c-ba38-9411c584464b\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " Oct 31 04:03:03.828068 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.828025 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-istio-token\" (UniqueName: \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-istio-token\") pod \"61524595-92d6-418c-ba38-9411c584464b\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " Oct 31 04:03:03.828376 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.828083 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-data\") pod \"61524595-92d6-418c-ba38-9411c584464b\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " Oct 31 04:03:03.829018 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.828906 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-credential-socket\") pod \"61524595-92d6-418c-ba38-9411c584464b\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " Oct 31 04:03:03.829177 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.829040 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61524595-92d6-418c-ba38-9411c584464b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61524595-92d6-418c-ba38-9411c584464b-istio-podinfo\") pod \"61524595-92d6-418c-ba38-9411c584464b\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " Oct 31 04:03:03.829251 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.829188 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-envoy\") pod \"61524595-92d6-418c-ba38-9411c584464b\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " Oct 31 04:03:03.829380 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.829361 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-kube-api-access-d6gs5\" (UniqueName: \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-kube-api-access-d6gs5\") pod \"61524595-92d6-418c-ba38-9411c584464b\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " Oct 31 04:03:03.829452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.829409 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61524595-92d6-418c-ba38-9411c584464b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61524595-92d6-418c-ba38-9411c584464b-istiod-ca-cert\") pod \"61524595-92d6-418c-ba38-9411c584464b\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " Oct 31 04:03:03.829576 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.829479 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-certs\") pod \"61524595-92d6-418c-ba38-9411c584464b\" (UID: \"61524595-92d6-418c-ba38-9411c584464b\") " Oct 31 04:03:03.830492 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.830462 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-credential-socket" pod "61524595-92d6-418c-ba38-9411c584464b" (UID: "61524595-92d6-418c-ba38-9411c584464b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:03.831092 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.831042 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-data" pod "61524595-92d6-418c-ba38-9411c584464b" (UID: "61524595-92d6-418c-ba38-9411c584464b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:03.831274 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.831105 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-certs" pod "61524595-92d6-418c-ba38-9411c584464b" (UID: "61524595-92d6-418c-ba38-9411c584464b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:03.831274 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.831175 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61524595-92d6-418c-ba38-9411c584464b-istiod-ca-cert" pod "61524595-92d6-418c-ba38-9411c584464b" (UID: "61524595-92d6-418c-ba38-9411c584464b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:03:03.831550 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.831521 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-socket" pod "61524595-92d6-418c-ba38-9411c584464b" (UID: "61524595-92d6-418c-ba38-9411c584464b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:03.833357 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.833324 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-envoy" pod "61524595-92d6-418c-ba38-9411c584464b" (UID: "61524595-92d6-418c-ba38-9411c584464b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:03:03.833618 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.833593 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-istio-token" pod "61524595-92d6-418c-ba38-9411c584464b" (UID: "61524595-92d6-418c-ba38-9411c584464b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:03.833888 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.833858 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-kube-api-access-d6gs5" pod "61524595-92d6-418c-ba38-9411c584464b" (UID: "61524595-92d6-418c-ba38-9411c584464b"). InnerVolumeSpecName "kube-api-access-d6gs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:03.834640 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.834605 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61524595-92d6-418c-ba38-9411c584464b-istio-podinfo" pod "61524595-92d6-418c-ba38-9411c584464b" (UID: "61524595-92d6-418c-ba38-9411c584464b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:03:03.932114 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.932068 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6gs5\" (UniqueName: \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-kube-api-access-d6gs5\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.932114 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.932112 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61524595-92d6-418c-ba38-9411c584464b-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.932114 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.932171 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.932114 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.932180 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.932492 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.932189 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61524595-92d6-418c-ba38-9411c584464b-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.932492 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.932206 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.932492 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.932215 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.932492 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.932227 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61524595-92d6-418c-ba38-9411c584464b-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.932492 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.932242 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61524595-92d6-418c-ba38-9411c584464b-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:03.964071 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.964038 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" Oct 31 04:03:03.964262 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.964038 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8" event={"ID":"5ef90cb4-6f28-437e-a875-80d202b3f16c","Type":"ContainerDied","Data":"da2c45697996e68eb436673e78ce23a13696506faeb64052a311536a22872fd5"} Oct 31 04:03:03.964262 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.964258 4426 scope.go:122] "RemoveContainer" containerID="9a198853c7f27a32c63e044fdff69d060af08f0465ea99c48a1fd48a2a45c5f1" Oct 31 04:03:03.966087 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.966062 4426 generic.go:356] "Generic (PLEG): container finished" podID="61524595-92d6-418c-ba38-9411c584464b" containerID="122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341" exitCode=0 Oct 31 04:03:03.966238 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.966140 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" Oct 31 04:03:03.966238 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.966188 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" event={"ID":"61524595-92d6-418c-ba38-9411c584464b","Type":"ContainerDied","Data":"122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341"} Oct 31 04:03:03.966348 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.966258 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs" event={"ID":"61524595-92d6-418c-ba38-9411c584464b","Type":"ContainerDied","Data":"6736a21a5cf6a7484b4e38061e15415dbc3ae0520c1e68b3d73ef5b729a071c4"} Oct 31 04:03:03.978139 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.978107 4426 scope.go:122] "RemoveContainer" containerID="122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341" Oct 31 04:03:03.991858 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.991828 4426 scope.go:122] "RemoveContainer" containerID="122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341" Oct 31 04:03:03.992379 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:03:03.992342 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341\": container with ID starting with 122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341 not found: ID does not exist" containerID="122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341" Oct 31 04:03:03.992481 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:03.992392 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341"} err="failed to get container status \"122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341\": rpc error: code = NotFound desc = could not find container \"122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341\": container with ID starting with 122a26159fa703b7ad7ad7488ff52796220006558c19482948a68f7de8b85341 not found: ID does not exist" Oct 31 04:03:04.122389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:04.122296 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8"] Oct 31 04:03:04.129827 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:04.129791 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-ldcw8"] Oct 31 04:03:04.232058 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:04.232027 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs"] Oct 31 04:03:04.245289 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:04.245249 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-2hhgs"] Oct 31 04:03:05.134411 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:05.134373 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" path="/var/lib/kubelet/pods/5ef90cb4-6f28-437e-a875-80d202b3f16c/volumes" Oct 31 04:03:05.135164 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:05.135111 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61524595-92d6-418c-ba38-9411c584464b" path="/var/lib/kubelet/pods/61524595-92d6-418c-ba38-9411c584464b/volumes" Oct 31 04:03:05.135779 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:05.135753 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a303237c-f769-4565-bb71-de2d23a0519b" path="/var/lib/kubelet/pods/a303237c-f769-4565-bb71-de2d23a0519b/volumes" Oct 31 04:03:05.136910 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:05.136887 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" path="/var/lib/kubelet/pods/b790d330-2c37-4ce4-8cf3-57afa5e12d63/volumes" Oct 31 04:03:08.617440 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.617405 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw"] Oct 31 04:03:08.619252 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.619217 4426 cpu_manager.go:414] "containerMap: removing container" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerName="istio-proxy" Oct 31 04:03:08.619470 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.619449 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerName="istio-proxy" Oct 31 04:03:08.619579 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.619568 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerName="cleanup" Oct 31 04:03:08.619672 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.619659 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerName="cleanup" Oct 31 04:03:08.619784 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.619769 4426 cpu_manager.go:414] "containerMap: removing container" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" Oct 31 04:03:08.619918 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.619904 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" Oct 31 04:03:08.620037 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.620025 4426 cpu_manager.go:414] "containerMap: removing container" podUID="a303237c-f769-4565-bb71-de2d23a0519b" containerName="istio-proxy" Oct 31 04:03:08.620119 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.620108 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a303237c-f769-4565-bb71-de2d23a0519b" containerName="istio-proxy" Oct 31 04:03:08.620261 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.620250 4426 cpu_manager.go:414] "containerMap: removing container" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerName="cleanup" Oct 31 04:03:08.620346 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.620336 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54338ebc-fe16-444d-9cfb-ff8537bb7c2e" containerName="cleanup" Oct 31 04:03:08.620440 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.620431 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerName="cleanup" Oct 31 04:03:08.620515 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.620505 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29acde0-cc51-41a1-91c4-7522b9c4978c" containerName="cleanup" Oct 31 04:03:08.620601 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.620591 4426 cpu_manager.go:414] "containerMap: removing container" podUID="06bb90ba-05f7-44c4-a917-35de0ac617dd" containerName="payload-pre-processing" Oct 31 04:03:08.620676 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.620665 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06bb90ba-05f7-44c4-a917-35de0ac617dd" containerName="payload-pre-processing" Oct 31 04:03:08.620767 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.620757 4426 cpu_manager.go:414] "containerMap: removing container" podUID="514aed0d-15b6-4569-9139-2362d7f0ca38" containerName="payload-processing" Oct 31 04:03:08.620860 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.620848 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514aed0d-15b6-4569-9139-2362d7f0ca38" containerName="payload-processing" Oct 31 04:03:08.620963 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.620952 4426 cpu_manager.go:414] "containerMap: removing container" podUID="52ffd0b4-21d4-4728-8b14-526aebb469dc" containerName="revoke-keys" Oct 31 04:03:08.621035 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621025 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ffd0b4-21d4-4728-8b14-526aebb469dc" containerName="revoke-keys" Oct 31 04:03:08.621155 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621143 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" Oct 31 04:03:08.621253 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621242 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" Oct 31 04:03:08.621568 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621535 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ffd0b4-21d4-4728-8b14-526aebb469dc" containerName="revoke-keys" Oct 31 04:03:08.621568 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621564 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f779ce49-773a-4bc7-87db-be2271baa5d0" containerName="cleanup" Oct 31 04:03:08.621780 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621577 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef90cb4-6f28-437e-a875-80d202b3f16c" containerName="istio-proxy" Oct 31 04:03:08.621780 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621593 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="b790d330-2c37-4ce4-8cf3-57afa5e12d63" containerName="istio-proxy" Oct 31 04:03:08.621780 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621603 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="61524595-92d6-418c-ba38-9411c584464b" containerName="istio-proxy" Oct 31 04:03:08.621780 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621615 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="a303237c-f769-4565-bb71-de2d23a0519b" containerName="istio-proxy" Oct 31 04:03:08.621780 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621627 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c118d9fb-524e-4287-919e-fd9a760bc4bc" containerName="cleanup" Oct 31 04:03:08.621780 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621639 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b647cae-10e2-44e2-8359-241944b5acfe" containerName="cleanup" Oct 31 04:03:08.621780 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621653 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="06bb90ba-05f7-44c4-a917-35de0ac617dd" containerName="payload-pre-processing" Oct 31 04:03:08.621780 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621666 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="514aed0d-15b6-4569-9139-2362d7f0ca38" containerName="payload-processing" Oct 31 04:03:08.621780 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.621677 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67baf37-cce6-4ce0-ae7a-f23571fd3ea4" containerName="cleanup" Oct 31 04:03:08.643037 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.643001 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:03:08.771151 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.745062 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqgn\" (UniqueName: \"kubernetes.io/projected/96e43cfe-4f33-4011-8588-eae8270d23cd-kube-api-access-5hqgn\") pod \"payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw\" (UID: \"96e43cfe-4f33-4011-8588-eae8270d23cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:03:08.800150 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.771677 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e43cfe-4f33-4011-8588-eae8270d23cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw\" (UID: \"96e43cfe-4f33-4011-8588-eae8270d23cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:03:08.806151 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.752791 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-288b63-dockercfg-t9rnk\"" Oct 31 04:03:08.806412 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.752625 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw"] Oct 31 04:03:08.810400 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.810348 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n"] Oct 31 04:03:08.818677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.778261 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-288b63\"" Oct 31 04:03:08.824475 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.824447 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:03:08.873200 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.873091 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n"] Oct 31 04:03:08.902530 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.902494 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e43cfe-4f33-4011-8588-eae8270d23cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw\" (UID: \"96e43cfe-4f33-4011-8588-eae8270d23cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:03:08.902726 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.902542 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c653c61a-7630-46e7-99e3-f0b48471b6a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n\" (UID: \"c653c61a-7630-46e7-99e3-f0b48471b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:03:08.902726 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.902583 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxvd5\" (UniqueName: \"kubernetes.io/projected/c653c61a-7630-46e7-99e3-f0b48471b6a1-kube-api-access-xxvd5\") pod \"payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n\" (UID: \"c653c61a-7630-46e7-99e3-f0b48471b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:03:08.902866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.902770 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqgn\" (UniqueName: \"kubernetes.io/projected/96e43cfe-4f33-4011-8588-eae8270d23cd-kube-api-access-5hqgn\") pod \"payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw\" (UID: \"96e43cfe-4f33-4011-8588-eae8270d23cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:03:08.903285 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.903258 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e43cfe-4f33-4011-8588-eae8270d23cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw\" (UID: \"96e43cfe-4f33-4011-8588-eae8270d23cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:03:08.960888 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:08.960855 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqgn\" (UniqueName: \"kubernetes.io/projected/96e43cfe-4f33-4011-8588-eae8270d23cd-kube-api-access-5hqgn\") pod \"payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw\" (UID: \"96e43cfe-4f33-4011-8588-eae8270d23cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:03:09.003833 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:09.003793 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxvd5\" (UniqueName: \"kubernetes.io/projected/c653c61a-7630-46e7-99e3-f0b48471b6a1-kube-api-access-xxvd5\") pod \"payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n\" (UID: \"c653c61a-7630-46e7-99e3-f0b48471b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:03:09.004172 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:09.004114 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c653c61a-7630-46e7-99e3-f0b48471b6a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n\" (UID: \"c653c61a-7630-46e7-99e3-f0b48471b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:03:09.005057 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:09.005033 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c653c61a-7630-46e7-99e3-f0b48471b6a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n\" (UID: \"c653c61a-7630-46e7-99e3-f0b48471b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:03:09.028013 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:09.027978 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:03:09.061878 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:09.061843 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxvd5\" (UniqueName: \"kubernetes.io/projected/c653c61a-7630-46e7-99e3-f0b48471b6a1-kube-api-access-xxvd5\") pod \"payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n\" (UID: \"c653c61a-7630-46e7-99e3-f0b48471b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:03:09.160573 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:09.160539 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:03:09.420414 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:09.420382 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw"] Oct 31 04:03:09.421726 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:03:09.421695 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e43cfe_4f33_4011_8588_eae8270d23cd.slice/crio-83fbc7232b46a48f33b4b86518e948a4f7a869cd75852574a44760ffee6f6c74 WatchSource:0}: Error finding container 83fbc7232b46a48f33b4b86518e948a4f7a869cd75852574a44760ffee6f6c74: Status 404 returned error can't find the container with id 83fbc7232b46a48f33b4b86518e948a4f7a869cd75852574a44760ffee6f6c74 Oct 31 04:03:09.551735 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:09.551700 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n"] Oct 31 04:03:09.555082 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:03:09.555053 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc653c61a_7630_46e7_99e3_f0b48471b6a1.slice/crio-772ab12ff51a122b93aa60a79db1052c6edb3fbcf49350ca4ba9be34a86426e8 WatchSource:0}: Error finding container 772ab12ff51a122b93aa60a79db1052c6edb3fbcf49350ca4ba9be34a86426e8: Status 404 returned error can't find the container with id 772ab12ff51a122b93aa60a79db1052c6edb3fbcf49350ca4ba9be34a86426e8 Oct 31 04:03:10.029816 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:10.029773 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" event={"ID":"96e43cfe-4f33-4011-8588-eae8270d23cd","Type":"ContainerStarted","Data":"348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce"} Oct 31 04:03:10.030278 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:10.029837 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" event={"ID":"96e43cfe-4f33-4011-8588-eae8270d23cd","Type":"ContainerStarted","Data":"83fbc7232b46a48f33b4b86518e948a4f7a869cd75852574a44760ffee6f6c74"} Oct 31 04:03:10.030278 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:10.029867 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:03:10.031664 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:10.031632 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" event={"ID":"c653c61a-7630-46e7-99e3-f0b48471b6a1","Type":"ContainerStarted","Data":"0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec"} Oct 31 04:03:10.031814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:10.031679 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" event={"ID":"c653c61a-7630-46e7-99e3-f0b48471b6a1","Type":"ContainerStarted","Data":"772ab12ff51a122b93aa60a79db1052c6edb3fbcf49350ca4ba9be34a86426e8"} Oct 31 04:03:10.031814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:10.031701 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:03:10.147979 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:10.147920 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" podStartSLOduration=2.147904608 podStartE2EDuration="2.147904608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:10.143245409 +0000 UTC m=+2087.143870636" watchObservedRunningTime="2026-10-31 04:03:10.147904608 +0000 UTC m=+2087.148529835" Oct 31 04:03:10.290818 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:10.290494 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" podStartSLOduration=2.290473491 podStartE2EDuration="2.290473491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:10.28966545 +0000 UTC m=+2087.290290698" watchObservedRunningTime="2026-10-31 04:03:10.290473491 +0000 UTC m=+2087.291098720" Oct 31 04:03:17.062610 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.062572 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g"] Oct 31 04:03:17.067359 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.067339 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:17.116882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.116847 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:03:17.144966 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.144930 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xx94\" (UniqueName: \"kubernetes.io/projected/40dbf523-aa55-42e0-a538-a0a5d3ae068a-kube-api-access-4xx94\") pod \"maas-api-revoke-sub-03173b5c-n6z4g\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:17.145231 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.145063 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03173b5c-n6z4g\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:17.145331 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.145245 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03173b5c-n6z4g\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:17.163459 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.163425 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:03:17.163632 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.163542 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:03:17.163632 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.163442 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:03:17.171176 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.171146 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g"] Oct 31 04:03:17.246718 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.246683 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03173b5c-n6z4g\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:17.246921 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.246789 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xx94\" (UniqueName: \"kubernetes.io/projected/40dbf523-aa55-42e0-a538-a0a5d3ae068a-kube-api-access-4xx94\") pod \"maas-api-revoke-sub-03173b5c-n6z4g\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:17.246921 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.246861 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03173b5c-n6z4g\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:17.247632 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.247602 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03173b5c-n6z4g\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:17.249753 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.249721 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03173b5c-n6z4g\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:17.305753 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.305714 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xx94\" (UniqueName: \"kubernetes.io/projected/40dbf523-aa55-42e0-a538-a0a5d3ae068a-kube-api-access-4xx94\") pod \"maas-api-revoke-sub-03173b5c-n6z4g\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:17.382804 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.382712 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:17.820025 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:17.819988 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g"] Oct 31 04:03:17.824939 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:03:17.824905 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dbf523_aa55_42e0_a538_a0a5d3ae068a.slice/crio-9613d8ed3b0f8556df87c6efa4e02eb36d206ae0346b142d9352902f8da94909 WatchSource:0}: Error finding container 9613d8ed3b0f8556df87c6efa4e02eb36d206ae0346b142d9352902f8da94909: Status 404 returned error can't find the container with id 9613d8ed3b0f8556df87c6efa4e02eb36d206ae0346b142d9352902f8da94909 Oct 31 04:03:18.105758 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:18.105669 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" event={"ID":"40dbf523-aa55-42e0-a538-a0a5d3ae068a","Type":"ContainerStarted","Data":"9613d8ed3b0f8556df87c6efa4e02eb36d206ae0346b142d9352902f8da94909"} Oct 31 04:03:19.115952 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:19.115858 4426 generic.go:356] "Generic (PLEG): container finished" podID="40dbf523-aa55-42e0-a538-a0a5d3ae068a" containerID="2e63bf5a33f4e057dd749747e93e6e5a0be7187b163fd138eb5f5c5f686c5e8d" exitCode=0 Oct 31 04:03:19.116399 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:19.115940 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" event={"ID":"40dbf523-aa55-42e0-a538-a0a5d3ae068a","Type":"ContainerDied","Data":"2e63bf5a33f4e057dd749747e93e6e5a0be7187b163fd138eb5f5c5f686c5e8d"} Oct 31 04:03:20.305438 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:20.305406 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:20.403350 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:20.403320 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-openshift-ca\") pod \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " Oct 31 04:03:20.403576 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:20.403399 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-xks-ca\") pod \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " Oct 31 04:03:20.403654 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:20.403646 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40dbf523-aa55-42e0-a538-a0a5d3ae068a-kube-api-access-4xx94\" (UniqueName: \"kubernetes.io/projected/40dbf523-aa55-42e0-a538-a0a5d3ae068a-kube-api-access-4xx94\") pod \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\" (UID: \"40dbf523-aa55-42e0-a538-a0a5d3ae068a\") " Oct 31 04:03:20.403769 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:20.403743 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-openshift-ca" pod "40dbf523-aa55-42e0-a538-a0a5d3ae068a" (UID: "40dbf523-aa55-42e0-a538-a0a5d3ae068a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:03:20.404117 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:20.404090 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:20.406102 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:20.405666 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-xks-ca" pod "40dbf523-aa55-42e0-a538-a0a5d3ae068a" (UID: "40dbf523-aa55-42e0-a538-a0a5d3ae068a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:03:20.406102 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:20.405843 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40dbf523-aa55-42e0-a538-a0a5d3ae068a-kube-api-access-4xx94" pod "40dbf523-aa55-42e0-a538-a0a5d3ae068a" (UID: "40dbf523-aa55-42e0-a538-a0a5d3ae068a"). InnerVolumeSpecName "kube-api-access-4xx94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:20.505316 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:20.505282 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xx94\" (UniqueName: \"kubernetes.io/projected/40dbf523-aa55-42e0-a538-a0a5d3ae068a-kube-api-access-4xx94\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:20.505316 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:20.505310 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40dbf523-aa55-42e0-a538-a0a5d3ae068a-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:21.046024 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:21.045994 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:03:21.046523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:21.046502 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:03:21.140558 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:21.140528 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" Oct 31 04:03:21.140751 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:21.140526 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g" event={"ID":"40dbf523-aa55-42e0-a538-a0a5d3ae068a","Type":"ContainerDied","Data":"9613d8ed3b0f8556df87c6efa4e02eb36d206ae0346b142d9352902f8da94909"} Oct 31 04:03:21.140821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:21.140785 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9613d8ed3b0f8556df87c6efa4e02eb36d206ae0346b142d9352902f8da94909" Oct 31 04:03:22.704910 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:22.704873 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:03:23.567501 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:23.567473 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:03:23.576241 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:23.576211 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:03:23.602385 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:23.602353 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf"] Oct 31 04:03:23.609206 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:23.609172 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-22e20532-fdnhf"] Oct 31 04:03:25.133335 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:25.133293 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ffd0b4-21d4-4728-8b14-526aebb469dc" path="/var/lib/kubelet/pods/52ffd0b4-21d4-4728-8b14-526aebb469dc/volumes" Oct 31 04:03:25.333844 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:25.333809 4426 scope.go:122] "RemoveContainer" containerID="f37f7c90f8e66159f1d4322baa37f936319cfe3178443cd4873c68c35e956653" Oct 31 04:03:25.344914 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:25.344892 4426 scope.go:122] "RemoveContainer" containerID="f06c7c0f80182b1f8743e008d78fa19999b0d6fb82cd4c946e664c011a48b7d8" Oct 31 04:03:25.354609 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:25.354578 4426 scope.go:122] "RemoveContainer" containerID="62754e8d71090d161e0f8d75ca4824057e0968de6baee2f8882734a5f17c20d7" Oct 31 04:03:25.363864 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:25.363839 4426 scope.go:122] "RemoveContainer" containerID="42ca271eda326be0e1a33ca86798d4f11122cbd9cbd9a669aee8e4cab7b740df" Oct 31 04:03:32.851118 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:03:32.851078 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:03:36.904702 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:36.904618 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp"] Oct 31 04:03:36.905429 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:36.905410 4426 cpu_manager.go:414] "containerMap: removing container" podUID="40dbf523-aa55-42e0-a538-a0a5d3ae068a" containerName="revoke-keys" Oct 31 04:03:36.905477 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:36.905430 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40dbf523-aa55-42e0-a538-a0a5d3ae068a" containerName="revoke-keys" Oct 31 04:03:36.905565 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:36.905556 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="40dbf523-aa55-42e0-a538-a0a5d3ae068a" containerName="revoke-keys" Oct 31 04:03:36.910659 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:36.910633 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:36.960577 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:36.960463 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:03:36.960577 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:36.960545 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:03:37.002481 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.002444 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c6n6\" (UniqueName: \"kubernetes.io/projected/e94997f9-71c2-4670-b680-15708a3b0ff0-kube-api-access-5c6n6\") pod \"maas-api-revoke-sub-6277d04c-mwvzp\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:37.002728 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.002700 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6277d04c-mwvzp\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:37.002829 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.002743 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6277d04c-mwvzp\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:37.006410 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.006377 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:03:37.006678 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.006658 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:03:37.014481 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.014452 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp"] Oct 31 04:03:37.104568 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.104530 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c6n6\" (UniqueName: \"kubernetes.io/projected/e94997f9-71c2-4670-b680-15708a3b0ff0-kube-api-access-5c6n6\") pod \"maas-api-revoke-sub-6277d04c-mwvzp\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:37.104745 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.104720 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6277d04c-mwvzp\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:37.104810 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.104774 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6277d04c-mwvzp\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:37.105487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.105463 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6277d04c-mwvzp\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:37.107024 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.107000 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6277d04c-mwvzp\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:37.163191 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.163098 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c6n6\" (UniqueName: \"kubernetes.io/projected/e94997f9-71c2-4670-b680-15708a3b0ff0-kube-api-access-5c6n6\") pod \"maas-api-revoke-sub-6277d04c-mwvzp\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:37.226457 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.226423 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:37.607265 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:37.607185 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp"] Oct 31 04:03:37.611532 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:03:37.611502 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94997f9_71c2_4670_b680_15708a3b0ff0.slice/crio-e7d8b27c3205210701da95d35ead7e78f8ca2436f573330e417a6803fbd206d6 WatchSource:0}: Error finding container e7d8b27c3205210701da95d35ead7e78f8ca2436f573330e417a6803fbd206d6: Status 404 returned error can't find the container with id e7d8b27c3205210701da95d35ead7e78f8ca2436f573330e417a6803fbd206d6 Oct 31 04:03:38.421146 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:38.420468 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" event={"ID":"e94997f9-71c2-4670-b680-15708a3b0ff0","Type":"ContainerStarted","Data":"c3fc8d86446fa2112c3329517d7f02ce39d7d88dd8559d7c8ae3569c004ff302"} Oct 31 04:03:38.421146 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:38.420573 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" event={"ID":"e94997f9-71c2-4670-b680-15708a3b0ff0","Type":"ContainerStarted","Data":"e7d8b27c3205210701da95d35ead7e78f8ca2436f573330e417a6803fbd206d6"} Oct 31 04:03:38.574740 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:38.574688 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" podStartSLOduration=2.574673126 podStartE2EDuration="2.574673126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:38.52928461 +0000 UTC m=+2115.529909839" watchObservedRunningTime="2026-10-31 04:03:38.574673126 +0000 UTC m=+2115.575298371" Oct 31 04:03:39.433493 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:39.433450 4426 generic.go:356] "Generic (PLEG): container finished" podID="e94997f9-71c2-4670-b680-15708a3b0ff0" containerID="c3fc8d86446fa2112c3329517d7f02ce39d7d88dd8559d7c8ae3569c004ff302" exitCode=0 Oct 31 04:03:39.434031 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:39.433572 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" event={"ID":"e94997f9-71c2-4670-b680-15708a3b0ff0","Type":"ContainerDied","Data":"c3fc8d86446fa2112c3329517d7f02ce39d7d88dd8559d7c8ae3569c004ff302"} Oct 31 04:03:40.636442 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:40.636408 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:40.764952 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:40.764860 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94997f9-71c2-4670-b680-15708a3b0ff0-kube-api-access-5c6n6\" (UniqueName: \"kubernetes.io/projected/e94997f9-71c2-4670-b680-15708a3b0ff0-kube-api-access-5c6n6\") pod \"e94997f9-71c2-4670-b680-15708a3b0ff0\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " Oct 31 04:03:40.765156 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:40.765066 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-openshift-ca\") pod \"e94997f9-71c2-4670-b680-15708a3b0ff0\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " Oct 31 04:03:40.765273 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:40.765248 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-xks-ca\") pod \"e94997f9-71c2-4670-b680-15708a3b0ff0\" (UID: \"e94997f9-71c2-4670-b680-15708a3b0ff0\") " Oct 31 04:03:40.765499 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:40.765468 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-openshift-ca" pod "e94997f9-71c2-4670-b680-15708a3b0ff0" (UID: "e94997f9-71c2-4670-b680-15708a3b0ff0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:03:40.766244 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:40.766214 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:40.767476 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:40.767445 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94997f9-71c2-4670-b680-15708a3b0ff0-kube-api-access-5c6n6" pod "e94997f9-71c2-4670-b680-15708a3b0ff0" (UID: "e94997f9-71c2-4670-b680-15708a3b0ff0"). InnerVolumeSpecName "kube-api-access-5c6n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:40.767898 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:40.767874 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-xks-ca" pod "e94997f9-71c2-4670-b680-15708a3b0ff0" (UID: "e94997f9-71c2-4670-b680-15708a3b0ff0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:03:40.867361 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:40.867327 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94997f9-71c2-4670-b680-15708a3b0ff0-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:40.867361 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:40.867355 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c6n6\" (UniqueName: \"kubernetes.io/projected/e94997f9-71c2-4670-b680-15708a3b0ff0-kube-api-access-5c6n6\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:41.457271 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:41.457170 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" Oct 31 04:03:41.457271 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:41.457182 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp" event={"ID":"e94997f9-71c2-4670-b680-15708a3b0ff0","Type":"ContainerDied","Data":"e7d8b27c3205210701da95d35ead7e78f8ca2436f573330e417a6803fbd206d6"} Oct 31 04:03:41.457271 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:41.457241 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7d8b27c3205210701da95d35ead7e78f8ca2436f573330e417a6803fbd206d6" Oct 31 04:03:41.909559 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:41.909521 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6"] Oct 31 04:03:41.910627 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:41.910603 4426 cpu_manager.go:414] "containerMap: removing container" podUID="e94997f9-71c2-4670-b680-15708a3b0ff0" containerName="revoke-keys" Oct 31 04:03:41.910673 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:41.910633 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94997f9-71c2-4670-b680-15708a3b0ff0" containerName="revoke-keys" Oct 31 04:03:41.910818 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:41.910805 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94997f9-71c2-4670-b680-15708a3b0ff0" containerName="revoke-keys" Oct 31 04:03:41.915426 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:41.915397 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:41.970321 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:41.970289 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6"] Oct 31 04:03:42.014207 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.008460 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:03:42.014207 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.008470 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:03:42.014207 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.008518 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:03:42.014207 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.011517 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:03:42.090384 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.090340 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:42.090559 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.090502 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ngmc\" (UniqueName: \"kubernetes.io/projected/7eaf0207-9a3d-449c-829f-d7437958dc5b-kube-api-access-2ngmc\") pod \"maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:42.090620 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.090598 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:42.191740 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.191625 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ngmc\" (UniqueName: \"kubernetes.io/projected/7eaf0207-9a3d-449c-829f-d7437958dc5b-kube-api-access-2ngmc\") pod \"maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:42.191740 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.191720 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:42.192007 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.191989 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:42.192471 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.192436 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:42.194398 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.194366 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:42.249463 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.249430 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ngmc\" (UniqueName: \"kubernetes.io/projected/7eaf0207-9a3d-449c-829f-d7437958dc5b-kube-api-access-2ngmc\") pod \"maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:42.531585 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.531489 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:42.925845 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:42.925812 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6"] Oct 31 04:03:42.929662 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:03:42.929635 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eaf0207_9a3d_449c_829f_d7437958dc5b.slice/crio-cd033e152a11ede7a1027f0bbdff9becf080d68906152c5eee13ced364ed05e8 WatchSource:0}: Error finding container cd033e152a11ede7a1027f0bbdff9becf080d68906152c5eee13ced364ed05e8: Status 404 returned error can't find the container with id cd033e152a11ede7a1027f0bbdff9becf080d68906152c5eee13ced364ed05e8 Oct 31 04:03:43.475561 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:43.475517 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" event={"ID":"7eaf0207-9a3d-449c-829f-d7437958dc5b","Type":"ContainerStarted","Data":"9b84ec2fe583df23479694bf14e0fbf68198afa76dd9c663417af5d283f08929"} Oct 31 04:03:43.475717 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:43.475581 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" event={"ID":"7eaf0207-9a3d-449c-829f-d7437958dc5b","Type":"ContainerStarted","Data":"cd033e152a11ede7a1027f0bbdff9becf080d68906152c5eee13ced364ed05e8"} Oct 31 04:03:43.635217 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:43.635098 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" podStartSLOduration=2.635084199 podStartE2EDuration="2.635084199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:43.583749254 +0000 UTC m=+2120.584374495" watchObservedRunningTime="2026-10-31 04:03:43.635084199 +0000 UTC m=+2120.635709427" Oct 31 04:03:44.485639 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:44.485604 4426 generic.go:356] "Generic (PLEG): container finished" podID="7eaf0207-9a3d-449c-829f-d7437958dc5b" containerID="9b84ec2fe583df23479694bf14e0fbf68198afa76dd9c663417af5d283f08929" exitCode=0 Oct 31 04:03:44.486074 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:44.485668 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" event={"ID":"7eaf0207-9a3d-449c-829f-d7437958dc5b","Type":"ContainerDied","Data":"9b84ec2fe583df23479694bf14e0fbf68198afa76dd9c663417af5d283f08929"} Oct 31 04:03:45.669287 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:45.669254 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:45.745593 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:45.745499 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-openshift-ca\") pod \"7eaf0207-9a3d-449c-829f-d7437958dc5b\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " Oct 31 04:03:45.745767 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:45.745607 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eaf0207-9a3d-449c-829f-d7437958dc5b-kube-api-access-2ngmc\" (UniqueName: \"kubernetes.io/projected/7eaf0207-9a3d-449c-829f-d7437958dc5b-kube-api-access-2ngmc\") pod \"7eaf0207-9a3d-449c-829f-d7437958dc5b\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " Oct 31 04:03:45.745860 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:45.745837 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-xks-ca\") pod \"7eaf0207-9a3d-449c-829f-d7437958dc5b\" (UID: \"7eaf0207-9a3d-449c-829f-d7437958dc5b\") " Oct 31 04:03:45.745957 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:45.745924 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-openshift-ca" pod "7eaf0207-9a3d-449c-829f-d7437958dc5b" (UID: "7eaf0207-9a3d-449c-829f-d7437958dc5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:03:45.746521 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:45.746494 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:45.748335 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:45.748295 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eaf0207-9a3d-449c-829f-d7437958dc5b-kube-api-access-2ngmc" pod "7eaf0207-9a3d-449c-829f-d7437958dc5b" (UID: "7eaf0207-9a3d-449c-829f-d7437958dc5b"). InnerVolumeSpecName "kube-api-access-2ngmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:45.748445 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:45.748361 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-xks-ca" pod "7eaf0207-9a3d-449c-829f-d7437958dc5b" (UID: "7eaf0207-9a3d-449c-829f-d7437958dc5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:03:45.848344 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:45.848303 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ngmc\" (UniqueName: \"kubernetes.io/projected/7eaf0207-9a3d-449c-829f-d7437958dc5b-kube-api-access-2ngmc\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:45.848344 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:45.848341 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eaf0207-9a3d-449c-829f-d7437958dc5b-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:46.311665 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:46.311630 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk"] Oct 31 04:03:46.321174 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:46.321137 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95"] Oct 31 04:03:46.358898 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:46.358860 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl"] Oct 31 04:03:46.597204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:46.596484 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" event={"ID":"7eaf0207-9a3d-449c-829f-d7437958dc5b","Type":"ContainerDied","Data":"cd033e152a11ede7a1027f0bbdff9becf080d68906152c5eee13ced364ed05e8"} Oct 31 04:03:46.597204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:46.596556 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd033e152a11ede7a1027f0bbdff9becf080d68906152c5eee13ced364ed05e8" Oct 31 04:03:46.597204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:46.596625 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6" Oct 31 04:03:50.786151 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:50.784252 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fxl7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:03:50.786151 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:50.784345 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:03:50.826287 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:50.826243 4426 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg5t2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:03:50.826605 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:50.826573 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg5t2" podUID="8b99c850-aaaf-4055-9a35-dc3ce08085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:03:51.092783 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.092689 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452"] Oct 31 04:03:51.094169 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.094142 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7eaf0207-9a3d-449c-829f-d7437958dc5b" containerName="revoke-keys" Oct 31 04:03:51.094319 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.094308 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eaf0207-9a3d-449c-829f-d7437958dc5b" containerName="revoke-keys" Oct 31 04:03:51.094597 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.094584 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eaf0207-9a3d-449c-829f-d7437958dc5b" containerName="revoke-keys" Oct 31 04:03:51.098756 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.098736 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:51.120721 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.120680 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhmk\" (UniqueName: \"kubernetes.io/projected/5064d4e6-b80b-4792-8fce-5284d5369af1-kube-api-access-gdhmk\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:51.120990 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.120967 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:51.134376 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.134328 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:51.189531 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.189499 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452"] Oct 31 04:03:51.194253 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.190706 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:03:51.194457 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.190753 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:03:51.198147 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.190819 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:03:51.219087 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.219044 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:03:51.322148 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.305200 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:51.331504 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.330113 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhmk\" (UniqueName: \"kubernetes.io/projected/5064d4e6-b80b-4792-8fce-5284d5369af1-kube-api-access-gdhmk\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:51.331504 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.330212 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:51.331504 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.310270 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:51.339295 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.339256 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:51.404970 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.404933 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhmk\" (UniqueName: \"kubernetes.io/projected/5064d4e6-b80b-4792-8fce-5284d5369af1-kube-api-access-gdhmk\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:51.438376 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.438338 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:51.836537 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:51.836422 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452"] Oct 31 04:03:51.841048 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:03:51.841011 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5064d4e6_b80b_4792_8fce_5284d5369af1.slice/crio-0ca6ed0bd600e07c8c8f513c918b5d2d121391519b09c65998a2265184f27476 WatchSource:0}: Error finding container 0ca6ed0bd600e07c8c8f513c918b5d2d121391519b09c65998a2265184f27476: Status 404 returned error can't find the container with id 0ca6ed0bd600e07c8c8f513c918b5d2d121391519b09c65998a2265184f27476 Oct 31 04:03:52.717033 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:52.716989 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" event={"ID":"5064d4e6-b80b-4792-8fce-5284d5369af1","Type":"ContainerStarted","Data":"fbe624bfce8764da44d394aa60a00b39646ff4b18f08505f70b2533eb6da1c34"} Oct 31 04:03:52.717261 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:52.717057 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" event={"ID":"5064d4e6-b80b-4792-8fce-5284d5369af1","Type":"ContainerStarted","Data":"0ca6ed0bd600e07c8c8f513c918b5d2d121391519b09c65998a2265184f27476"} Oct 31 04:03:52.829076 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:52.829010 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" podStartSLOduration=1.82899487 podStartE2EDuration="1.82899487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:52.825580375 +0000 UTC m=+2129.826205604" watchObservedRunningTime="2026-10-31 04:03:52.82899487 +0000 UTC m=+2129.829620097" Oct 31 04:03:53.728209 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:53.728161 4426 generic.go:356] "Generic (PLEG): container finished" podID="5064d4e6-b80b-4792-8fce-5284d5369af1" containerID="fbe624bfce8764da44d394aa60a00b39646ff4b18f08505f70b2533eb6da1c34" exitCode=0 Oct 31 04:03:53.728757 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:53.728250 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" event={"ID":"5064d4e6-b80b-4792-8fce-5284d5369af1","Type":"ContainerDied","Data":"fbe624bfce8764da44d394aa60a00b39646ff4b18f08505f70b2533eb6da1c34"} Oct 31 04:03:53.764898 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:53.764865 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck"] Oct 31 04:03:54.964697 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:54.964668 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:54.999200 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:54.998251 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5064d4e6-b80b-4792-8fce-5284d5369af1-kube-api-access-gdhmk\" (UniqueName: \"kubernetes.io/projected/5064d4e6-b80b-4792-8fce-5284d5369af1-kube-api-access-gdhmk\") pod \"5064d4e6-b80b-4792-8fce-5284d5369af1\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " Oct 31 04:03:54.999200 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:54.998348 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-openshift-ca\") pod \"5064d4e6-b80b-4792-8fce-5284d5369af1\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " Oct 31 04:03:54.999200 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:54.998469 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-xks-ca\") pod \"5064d4e6-b80b-4792-8fce-5284d5369af1\" (UID: \"5064d4e6-b80b-4792-8fce-5284d5369af1\") " Oct 31 04:03:54.999200 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:54.999036 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-openshift-ca" pod "5064d4e6-b80b-4792-8fce-5284d5369af1" (UID: "5064d4e6-b80b-4792-8fce-5284d5369af1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:03:55.001724 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:55.001692 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-xks-ca" pod "5064d4e6-b80b-4792-8fce-5284d5369af1" (UID: "5064d4e6-b80b-4792-8fce-5284d5369af1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:03:55.002582 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:55.002557 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5064d4e6-b80b-4792-8fce-5284d5369af1-kube-api-access-gdhmk" pod "5064d4e6-b80b-4792-8fce-5284d5369af1" (UID: "5064d4e6-b80b-4792-8fce-5284d5369af1"). InnerVolumeSpecName "kube-api-access-gdhmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:55.100499 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:55.100458 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdhmk\" (UniqueName: \"kubernetes.io/projected/5064d4e6-b80b-4792-8fce-5284d5369af1-kube-api-access-gdhmk\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:55.100499 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:55.100493 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:55.100499 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:55.100508 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5064d4e6-b80b-4792-8fce-5284d5369af1-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:03:55.750012 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:55.749974 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" event={"ID":"5064d4e6-b80b-4792-8fce-5284d5369af1","Type":"ContainerDied","Data":"0ca6ed0bd600e07c8c8f513c918b5d2d121391519b09c65998a2265184f27476"} Oct 31 04:03:55.750240 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:55.750037 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca6ed0bd600e07c8c8f513c918b5d2d121391519b09c65998a2265184f27476" Oct 31 04:03:55.750240 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:55.749982 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452" Oct 31 04:03:58.765711 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:03:58.765664 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" podUID="63c7ff47-3bdc-4426-b49a-f36bb921d757" containerName="payload-processing" containerID="cri-o://06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205" gracePeriod=25 Oct 31 04:04:01.208235 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.186705 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" podUID="63c7ff47-3bdc-4426-b49a-f36bb921d757" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:04:01.335954 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.335753 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" containerID="cri-o://fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5" gracePeriod=45 Oct 31 04:04:01.344377 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.344006 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" containerID="cri-o://f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75" gracePeriod=45 Oct 31 04:04:01.369376 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.369342 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": dial tcp 10.133.0.109:8000: connect: connection refused" Oct 31 04:04:01.382991 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.382956 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" containerID="cri-o://2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284" gracePeriod=45 Oct 31 04:04:01.405551 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.405427 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": dial tcp 10.133.0.115:8000: connect: connection refused" Oct 31 04:04:01.703029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.702990 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 04:04:01.746754 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.746723 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 04:04:01.750048 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.750016 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 04:04:01.822417 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.820723 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38467302-482f-4d7a-a23d-a52d5bf56c44-kube-api-access-llmf5\" (UniqueName: \"kubernetes.io/projected/38467302-482f-4d7a-a23d-a52d5bf56c44-kube-api-access-llmf5\") pod \"38467302-482f-4d7a-a23d-a52d5bf56c44\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " Oct 31 04:04:01.823070 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823043 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-home\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-home\") pod \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " Oct 31 04:04:01.823272 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823255 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33640080-5615-4d98-bbcb-83fa262c3fe6-tls-certs\" (UniqueName: \"kubernetes.io/secret/33640080-5615-4d98-bbcb-83fa262c3fe6-tls-certs\") pod \"33640080-5615-4d98-bbcb-83fa262c3fe6\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " Oct 31 04:04:01.823406 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823393 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-tmp-dir\") pod \"33640080-5615-4d98-bbcb-83fa262c3fe6\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " Oct 31 04:04:01.823553 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823524 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-model-cache\") pod \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " Oct 31 04:04:01.823701 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823623 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-dshm\") pod \"33640080-5615-4d98-bbcb-83fa262c3fe6\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " Oct 31 04:04:01.823701 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823653 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-home\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-home\") pod \"33640080-5615-4d98-bbcb-83fa262c3fe6\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " Oct 31 04:04:01.823701 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823691 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-tmp-dir\") pod \"38467302-482f-4d7a-a23d-a52d5bf56c44\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " Oct 31 04:04:01.823863 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823768 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-kube-api-access-blq4j\" (UniqueName: \"kubernetes.io/projected/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-kube-api-access-blq4j\") pod \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " Oct 31 04:04:01.823863 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823824 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-model-cache\") pod \"38467302-482f-4d7a-a23d-a52d5bf56c44\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " Oct 31 04:04:01.823968 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823866 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tmp-dir\") pod \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " Oct 31 04:04:01.823968 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823913 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-home\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-home\") pod \"38467302-482f-4d7a-a23d-a52d5bf56c44\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " Oct 31 04:04:01.824075 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.823967 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38467302-482f-4d7a-a23d-a52d5bf56c44-tls-certs\" (UniqueName: \"kubernetes.io/secret/38467302-482f-4d7a-a23d-a52d5bf56c44-tls-certs\") pod \"38467302-482f-4d7a-a23d-a52d5bf56c44\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " Oct 31 04:04:01.824075 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.824000 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33640080-5615-4d98-bbcb-83fa262c3fe6-kube-api-access-g5bfx\" (UniqueName: \"kubernetes.io/projected/33640080-5615-4d98-bbcb-83fa262c3fe6-kube-api-access-g5bfx\") pod \"33640080-5615-4d98-bbcb-83fa262c3fe6\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " Oct 31 04:04:01.824075 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.824048 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-dshm\") pod \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " Oct 31 04:04:01.824310 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.824078 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-dshm\") pod \"38467302-482f-4d7a-a23d-a52d5bf56c44\" (UID: \"38467302-482f-4d7a-a23d-a52d5bf56c44\") " Oct 31 04:04:01.824310 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.824107 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-model-cache\") pod \"33640080-5615-4d98-bbcb-83fa262c3fe6\" (UID: \"33640080-5615-4d98-bbcb-83fa262c3fe6\") " Oct 31 04:04:01.824310 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.824252 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tls-certs\") pod \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\" (UID: \"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565\") " Oct 31 04:04:01.829972 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.829771 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tmp-dir" pod "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" (UID: "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.830474 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.830254 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-model-cache" pod "38467302-482f-4d7a-a23d-a52d5bf56c44" (UID: "38467302-482f-4d7a-a23d-a52d5bf56c44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.830926 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.830635 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-home" pod "33640080-5615-4d98-bbcb-83fa262c3fe6" (UID: "33640080-5615-4d98-bbcb-83fa262c3fe6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.831315 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.831293 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-model-cache" pod "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" (UID: "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.831905 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.831884 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-home" pod "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" (UID: "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.838083 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.837869 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-home" pod "38467302-482f-4d7a-a23d-a52d5bf56c44" (UID: "38467302-482f-4d7a-a23d-a52d5bf56c44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.838265 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.838112 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-tmp-dir" pod "33640080-5615-4d98-bbcb-83fa262c3fe6" (UID: "33640080-5615-4d98-bbcb-83fa262c3fe6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.838438 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.838402 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-tmp-dir" pod "38467302-482f-4d7a-a23d-a52d5bf56c44" (UID: "38467302-482f-4d7a-a23d-a52d5bf56c44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.839396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.838507 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-model-cache" pod "33640080-5615-4d98-bbcb-83fa262c3fe6" (UID: "33640080-5615-4d98-bbcb-83fa262c3fe6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.839396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.838652 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-dshm" pod "38467302-482f-4d7a-a23d-a52d5bf56c44" (UID: "38467302-482f-4d7a-a23d-a52d5bf56c44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.839396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.839006 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-dshm" pod "33640080-5615-4d98-bbcb-83fa262c3fe6" (UID: "33640080-5615-4d98-bbcb-83fa262c3fe6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.839650 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.839629 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33640080-5615-4d98-bbcb-83fa262c3fe6-tls-certs" pod "33640080-5615-4d98-bbcb-83fa262c3fe6" (UID: "33640080-5615-4d98-bbcb-83fa262c3fe6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:04:01.854595 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.854553 4426 generic.go:356] "Generic (PLEG): container finished" podID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerID="2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284" exitCode=0 Oct 31 04:04:01.854779 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.854757 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" event={"ID":"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565","Type":"ContainerDied","Data":"2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284"} Oct 31 04:04:01.854842 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.854798 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" event={"ID":"6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565","Type":"ContainerDied","Data":"e934a7d2a67fc5b1f37d7d149e6272eb578eecfa3ea3cf77df44ec3ee4aae4b6"} Oct 31 04:04:01.854842 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.854823 4426 scope.go:122] "RemoveContainer" containerID="2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284" Oct 31 04:04:01.855000 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.854985 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" Oct 31 04:04:01.858009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.857547 4426 generic.go:356] "Generic (PLEG): container finished" podID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerID="f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75" exitCode=0 Oct 31 04:04:01.858009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.857770 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" event={"ID":"38467302-482f-4d7a-a23d-a52d5bf56c44","Type":"ContainerDied","Data":"f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75"} Oct 31 04:04:01.858009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.857802 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" event={"ID":"38467302-482f-4d7a-a23d-a52d5bf56c44","Type":"ContainerDied","Data":"341c22da841854f449e82d58784bbd7571051adafb1bf5817743998ed2c02b0d"} Oct 31 04:04:01.858009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.857871 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95" Oct 31 04:04:01.860606 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.860247 4426 generic.go:356] "Generic (PLEG): container finished" podID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerID="fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5" exitCode=0 Oct 31 04:04:01.860606 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.860378 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" event={"ID":"33640080-5615-4d98-bbcb-83fa262c3fe6","Type":"ContainerDied","Data":"fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5"} Oct 31 04:04:01.860606 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.860408 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" event={"ID":"33640080-5615-4d98-bbcb-83fa262c3fe6","Type":"ContainerDied","Data":"fbeeae1aa1dd3be91a8918157a6a14d129fd45c4a6b497aff7cb29c9c85828e0"} Oct 31 04:04:01.860606 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.860468 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk" Oct 31 04:04:01.860949 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.860904 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38467302-482f-4d7a-a23d-a52d5bf56c44-kube-api-access-llmf5" pod "38467302-482f-4d7a-a23d-a52d5bf56c44" (UID: "38467302-482f-4d7a-a23d-a52d5bf56c44"). InnerVolumeSpecName "kube-api-access-llmf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:01.861231 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.861198 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tls-certs" pod "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" (UID: "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:04:01.863009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.862958 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38467302-482f-4d7a-a23d-a52d5bf56c44-tls-certs" pod "38467302-482f-4d7a-a23d-a52d5bf56c44" (UID: "38467302-482f-4d7a-a23d-a52d5bf56c44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:04:01.869809 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.869749 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33640080-5615-4d98-bbcb-83fa262c3fe6-kube-api-access-g5bfx" pod "33640080-5615-4d98-bbcb-83fa262c3fe6" (UID: "33640080-5615-4d98-bbcb-83fa262c3fe6"). InnerVolumeSpecName "kube-api-access-g5bfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:01.872087 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.871602 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-dshm" pod "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" (UID: "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:01.876718 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.876037 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-kube-api-access-blq4j" pod "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" (UID: "6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565"). InnerVolumeSpecName "kube-api-access-blq4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:01.876718 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.876314 4426 scope.go:122] "RemoveContainer" containerID="2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284" Oct 31 04:04:01.877485 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:01.877328 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284\": container with ID starting with 2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284 not found: ID does not exist" containerID="2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284" Oct 31 04:04:01.877485 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.877372 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284"} err="failed to get container status \"2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284\": rpc error: code = NotFound desc = could not find container \"2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284\": container with ID starting with 2d80a07896ecfda654a08fe1ba2a3bec5f7e4be245c8ae37296944a44b38a284 not found: ID does not exist" Oct 31 04:04:01.877485 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.877402 4426 scope.go:122] "RemoveContainer" containerID="f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75" Oct 31 04:04:01.894161 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.894135 4426 scope.go:122] "RemoveContainer" containerID="f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75" Oct 31 04:04:01.894558 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:01.894531 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75\": container with ID starting with f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75 not found: ID does not exist" containerID="f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75" Oct 31 04:04:01.894644 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.894571 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75"} err="failed to get container status \"f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75\": rpc error: code = NotFound desc = could not find container \"f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75\": container with ID starting with f34f84d0982d2a9386f58ddd2dd8cc8659cc020a7dd705fd5a58510c2386ad75 not found: ID does not exist" Oct 31 04:04:01.894644 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.894598 4426 scope.go:122] "RemoveContainer" containerID="fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5" Oct 31 04:04:01.906582 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.906558 4426 scope.go:122] "RemoveContainer" containerID="fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5" Oct 31 04:04:01.906941 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:01.906911 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5\": container with ID starting with fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5 not found: ID does not exist" containerID="fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5" Oct 31 04:04:01.907075 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.906943 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5"} err="failed to get container status \"fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5\": rpc error: code = NotFound desc = could not find container \"fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5\": container with ID starting with fbf4f7e2f5fd33a05dcd7d1fed3fcf8a9aa01d34a0f5bbd2a0988c173876bac5 not found: ID does not exist" Oct 31 04:04:01.927115 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927072 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927115 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927112 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927141 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927155 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927170 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blq4j\" (UniqueName: \"kubernetes.io/projected/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-kube-api-access-blq4j\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927186 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927199 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927214 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927226 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38467302-482f-4d7a-a23d-a52d5bf56c44-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927239 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5bfx\" (UniqueName: \"kubernetes.io/projected/33640080-5615-4d98-bbcb-83fa262c3fe6-kube-api-access-g5bfx\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927253 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927267 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38467302-482f-4d7a-a23d-a52d5bf56c44-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927281 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927295 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927310 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llmf5\" (UniqueName: \"kubernetes.io/projected/38467302-482f-4d7a-a23d-a52d5bf56c44-kube-api-access-llmf5\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927324 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927338 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33640080-5615-4d98-bbcb-83fa262c3fe6-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:01.927389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:01.927351 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33640080-5615-4d98-bbcb-83fa262c3fe6-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:02.216356 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:02.216299 4426 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b78aebf_ac0b_4b51_a1a4_4d7e11a4e565.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33640080_5615_4d98_bbcb_83fa262c3fe6.slice/crio-fbeeae1aa1dd3be91a8918157a6a14d129fd45c4a6b497aff7cb29c9c85828e0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38467302_482f_4d7a_a23d_a52d5bf56c44.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38467302_482f_4d7a_a23d_a52d5bf56c44.slice/crio-341c22da841854f449e82d58784bbd7571051adafb1bf5817743998ed2c02b0d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33640080_5615_4d98_bbcb_83fa262c3fe6.slice\": RecentStats: unable to find data in memory cache]" Oct 31 04:04:02.299314 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:02.299280 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95"] Oct 31 04:04:02.304004 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:02.303973 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-w4-68aa3e-kserve-7d4658b8f7-r8k95"] Oct 31 04:04:02.405279 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:02.405237 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk"] Oct 31 04:04:02.410339 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:02.410307 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/distinct-2-w4-68aa3e-kserve-67949648bb-7vglk"] Oct 31 04:04:02.515276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:02.515174 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl"] Oct 31 04:04:02.520071 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:02.520040 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl"] Oct 31 04:04:02.692161 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:02.692061 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-68aa3e/premium-w4-68aa3e-kserve-75d4d9cc5d-j85wl" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:04:02.851030 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:02.850936 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:04:03.132071 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:03.131990 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" path="/var/lib/kubelet/pods/33640080-5615-4d98-bbcb-83fa262c3fe6/volumes" Oct 31 04:04:03.132556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:03.132539 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" path="/var/lib/kubelet/pods/38467302-482f-4d7a-a23d-a52d5bf56c44/volumes" Oct 31 04:04:03.132953 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:03.132937 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" path="/var/lib/kubelet/pods/6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565/volumes" Oct 31 04:04:03.488844 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:03.488803 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62"] Oct 31 04:04:03.589930 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:03.589890 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4"] Oct 31 04:04:08.489857 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.489789 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" podUID="e2851d33-a4f5-4f39-b19a-990a2dd277dd" containerName="payload-processing" containerID="cri-o://3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8" gracePeriod=25 Oct 31 04:04:08.577559 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:04:08.577516 4426 logging.go:55] [core] [Channel #4521 SubChannel #4523]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.174:50052", ServerName: "10.133.0.174:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.174:50052: connect: connection refused" Oct 31 04:04:08.591224 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.591182 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" podUID="f2ea1ec7-6b32-43ab-9738-3deaf786a844" containerName="payload-pre-processing" containerID="cri-o://93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea" gracePeriod=25 Oct 31 04:04:08.807647 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.807611 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 04:04:08.882806 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.882779 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 04:04:08.941003 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.940965 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2851d33-a4f5-4f39-b19a-990a2dd277dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2851d33-a4f5-4f39-b19a-990a2dd277dd-plugins-config-volume\") pod \"e2851d33-a4f5-4f39-b19a-990a2dd277dd\" (UID: \"e2851d33-a4f5-4f39-b19a-990a2dd277dd\") " Oct 31 04:04:08.941230 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.941097 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2851d33-a4f5-4f39-b19a-990a2dd277dd-kube-api-access-jz64f\" (UniqueName: \"kubernetes.io/projected/e2851d33-a4f5-4f39-b19a-990a2dd277dd-kube-api-access-jz64f\") pod \"e2851d33-a4f5-4f39-b19a-990a2dd277dd\" (UID: \"e2851d33-a4f5-4f39-b19a-990a2dd277dd\") " Oct 31 04:04:08.941450 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.941420 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2851d33-a4f5-4f39-b19a-990a2dd277dd-plugins-config-volume" pod "e2851d33-a4f5-4f39-b19a-990a2dd277dd" (UID: "e2851d33-a4f5-4f39-b19a-990a2dd277dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:08.941732 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.941715 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2851d33-a4f5-4f39-b19a-990a2dd277dd-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:08.943840 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.943811 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2851d33-a4f5-4f39-b19a-990a2dd277dd-kube-api-access-jz64f" pod "e2851d33-a4f5-4f39-b19a-990a2dd277dd" (UID: "e2851d33-a4f5-4f39-b19a-990a2dd277dd"). InnerVolumeSpecName "kube-api-access-jz64f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:08.990447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.990412 4426 generic.go:356] "Generic (PLEG): container finished" podID="f2ea1ec7-6b32-43ab-9738-3deaf786a844" containerID="93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea" exitCode=0 Oct 31 04:04:08.990638 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.990455 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" Oct 31 04:04:08.990638 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.990494 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" event={"ID":"f2ea1ec7-6b32-43ab-9738-3deaf786a844","Type":"ContainerDied","Data":"93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea"} Oct 31 04:04:08.990638 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.990543 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4" event={"ID":"f2ea1ec7-6b32-43ab-9738-3deaf786a844","Type":"ContainerDied","Data":"daac39d29cf0253cb49ab03e9e8174aeb5e4a465277f9a1ad45b26a70de5f06f"} Oct 31 04:04:08.990638 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.990564 4426 scope.go:122] "RemoveContainer" containerID="93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea" Oct 31 04:04:08.991918 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.991889 4426 generic.go:356] "Generic (PLEG): container finished" podID="e2851d33-a4f5-4f39-b19a-990a2dd277dd" containerID="3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8" exitCode=0 Oct 31 04:04:08.992029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.991923 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" event={"ID":"e2851d33-a4f5-4f39-b19a-990a2dd277dd","Type":"ContainerDied","Data":"3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8"} Oct 31 04:04:08.992029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.991969 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" event={"ID":"e2851d33-a4f5-4f39-b19a-990a2dd277dd","Type":"ContainerDied","Data":"daf7bc0cc6e29b463e34e8cb55ebe07984f5b226bfd46afddd068923a11f27f4"} Oct 31 04:04:08.992029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:08.991936 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" Oct 31 04:04:09.003366 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.003345 4426 scope.go:122] "RemoveContainer" containerID="93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea" Oct 31 04:04:09.003711 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:09.003690 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea\": container with ID starting with 93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea not found: ID does not exist" containerID="93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea" Oct 31 04:04:09.003757 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.003719 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea"} err="failed to get container status \"93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea\": rpc error: code = NotFound desc = could not find container \"93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea\": container with ID starting with 93173df396c7f5cbb043df0d7068cd16b74e4ff95504c948b98c8fc4161a34ea not found: ID does not exist" Oct 31 04:04:09.003757 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.003738 4426 scope.go:122] "RemoveContainer" containerID="3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8" Oct 31 04:04:09.014585 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.014567 4426 scope.go:122] "RemoveContainer" containerID="3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8" Oct 31 04:04:09.014853 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:09.014837 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8\": container with ID starting with 3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8 not found: ID does not exist" containerID="3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8" Oct 31 04:04:09.014904 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.014859 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8"} err="failed to get container status \"3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8\": rpc error: code = NotFound desc = could not find container \"3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8\": container with ID starting with 3341625a40e23eec0bb7b621bf99c6975a431136d86bead119a5b0df3c8734f8 not found: ID does not exist" Oct 31 04:04:09.043052 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.043015 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ea1ec7-6b32-43ab-9738-3deaf786a844-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ea1ec7-6b32-43ab-9738-3deaf786a844-plugins-config-volume\") pod \"f2ea1ec7-6b32-43ab-9738-3deaf786a844\" (UID: \"f2ea1ec7-6b32-43ab-9738-3deaf786a844\") " Oct 31 04:04:09.043267 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.043188 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ea1ec7-6b32-43ab-9738-3deaf786a844-kube-api-access-6mznf\" (UniqueName: \"kubernetes.io/projected/f2ea1ec7-6b32-43ab-9738-3deaf786a844-kube-api-access-6mznf\") pod \"f2ea1ec7-6b32-43ab-9738-3deaf786a844\" (UID: \"f2ea1ec7-6b32-43ab-9738-3deaf786a844\") " Oct 31 04:04:09.043439 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.043416 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ea1ec7-6b32-43ab-9738-3deaf786a844-plugins-config-volume" pod "f2ea1ec7-6b32-43ab-9738-3deaf786a844" (UID: "f2ea1ec7-6b32-43ab-9738-3deaf786a844"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:09.043747 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.043728 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ea1ec7-6b32-43ab-9738-3deaf786a844-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:09.043747 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.043747 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz64f\" (UniqueName: \"kubernetes.io/projected/e2851d33-a4f5-4f39-b19a-990a2dd277dd-kube-api-access-jz64f\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:09.045695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.045665 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ea1ec7-6b32-43ab-9738-3deaf786a844-kube-api-access-6mznf" pod "f2ea1ec7-6b32-43ab-9738-3deaf786a844" (UID: "f2ea1ec7-6b32-43ab-9738-3deaf786a844"). InnerVolumeSpecName "kube-api-access-6mznf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:09.144866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.144834 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mznf\" (UniqueName: \"kubernetes.io/projected/f2ea1ec7-6b32-43ab-9738-3deaf786a844-kube-api-access-6mznf\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:09.185940 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.185906 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62"] Oct 31 04:04:09.190701 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.190672 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62"] Oct 31 04:04:09.402173 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.402120 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4"] Oct 31 04:04:09.407908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.407871 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4a3a5-f59f6b9c9-n8fq4"] Oct 31 04:04:09.577402 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:09.577360 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4a3a5-9b96cf667-phf62" podUID="e2851d33-a4f5-4f39-b19a-990a2dd277dd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.174:50052\" within 1s: context deadline exceeded" Oct 31 04:04:11.132648 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:11.132618 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2851d33-a4f5-4f39-b19a-990a2dd277dd" path="/var/lib/kubelet/pods/e2851d33-a4f5-4f39-b19a-990a2dd277dd/volumes" Oct 31 04:04:11.133079 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:11.133064 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ea1ec7-6b32-43ab-9738-3deaf786a844" path="/var/lib/kubelet/pods/f2ea1ec7-6b32-43ab-9738-3deaf786a844/volumes" Oct 31 04:04:11.171301 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:11.171254 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" podUID="63c7ff47-3bdc-4426-b49a-f36bb921d757" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:04:16.445063 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445017 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t"] Oct 31 04:04:16.445807 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445788 4426 cpu_manager.go:414] "containerMap: removing container" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" Oct 31 04:04:16.445875 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445810 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" Oct 31 04:04:16.445875 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445822 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" Oct 31 04:04:16.445875 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445828 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" Oct 31 04:04:16.445875 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445836 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f2ea1ec7-6b32-43ab-9738-3deaf786a844" containerName="payload-pre-processing" Oct 31 04:04:16.445875 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445842 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ea1ec7-6b32-43ab-9738-3deaf786a844" containerName="payload-pre-processing" Oct 31 04:04:16.445875 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445872 4426 cpu_manager.go:414] "containerMap: removing container" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" Oct 31 04:04:16.445875 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445877 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" Oct 31 04:04:16.446097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445885 4426 cpu_manager.go:414] "containerMap: removing container" podUID="e2851d33-a4f5-4f39-b19a-990a2dd277dd" containerName="payload-processing" Oct 31 04:04:16.446097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445892 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2851d33-a4f5-4f39-b19a-990a2dd277dd" containerName="payload-processing" Oct 31 04:04:16.446097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445927 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5064d4e6-b80b-4792-8fce-5284d5369af1" containerName="revoke-keys" Oct 31 04:04:16.446097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.445933 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5064d4e6-b80b-4792-8fce-5284d5369af1" containerName="revoke-keys" Oct 31 04:04:16.446097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.446042 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="38467302-482f-4d7a-a23d-a52d5bf56c44" containerName="main" Oct 31 04:04:16.446097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.446052 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2851d33-a4f5-4f39-b19a-990a2dd277dd" containerName="payload-processing" Oct 31 04:04:16.446097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.446062 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="5064d4e6-b80b-4792-8fce-5284d5369af1" containerName="revoke-keys" Oct 31 04:04:16.446097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.446071 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="33640080-5615-4d98-bbcb-83fa262c3fe6" containerName="main" Oct 31 04:04:16.446097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.446076 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b78aebf-ac0b-4b51-a1a4-4d7e11a4e565" containerName="main" Oct 31 04:04:16.446097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.446082 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ea1ec7-6b32-43ab-9738-3deaf786a844" containerName="payload-pre-processing" Oct 31 04:04:16.451536 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.451515 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:16.510837 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.510799 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:04:16.511001 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.510864 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:04:16.547819 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.547783 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:16.547999 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.547828 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lptnc\" (UniqueName: \"kubernetes.io/projected/c76d973c-8236-442a-9ef4-7c0564a09bd4-kube-api-access-lptnc\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:16.547999 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.547931 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:16.557564 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.557531 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:04:16.557788 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.557646 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:04:16.563651 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.563624 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t"] Oct 31 04:04:16.649399 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.649364 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:16.649602 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.649414 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lptnc\" (UniqueName: \"kubernetes.io/projected/c76d973c-8236-442a-9ef4-7c0564a09bd4-kube-api-access-lptnc\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:16.649602 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.649445 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:16.650300 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.650268 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:16.652140 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.652100 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:16.714211 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.710564 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lptnc\" (UniqueName: \"kubernetes.io/projected/c76d973c-8236-442a-9ef4-7c0564a09bd4-kube-api-access-lptnc\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:16.766936 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:16.766904 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:17.146118 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:17.146081 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t"] Oct 31 04:04:17.150386 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:04:17.150352 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76d973c_8236_442a_9ef4_7c0564a09bd4.slice/crio-21b7fca64a958c8310ee31638a1dc0821b28c809f1fb8510a6b4bd0b08fb2544 WatchSource:0}: Error finding container 21b7fca64a958c8310ee31638a1dc0821b28c809f1fb8510a6b4bd0b08fb2544: Status 404 returned error can't find the container with id 21b7fca64a958c8310ee31638a1dc0821b28c809f1fb8510a6b4bd0b08fb2544 Oct 31 04:04:18.123865 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:18.123830 4426 generic.go:356] "Generic (PLEG): container finished" podID="c76d973c-8236-442a-9ef4-7c0564a09bd4" containerID="7e46679c5e9c8e5fdbb885952cc1dead2e226f7f7f0e13b19eaf9820ad7c0c0b" exitCode=0 Oct 31 04:04:18.124481 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:18.124206 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" event={"ID":"c76d973c-8236-442a-9ef4-7c0564a09bd4","Type":"ContainerDied","Data":"7e46679c5e9c8e5fdbb885952cc1dead2e226f7f7f0e13b19eaf9820ad7c0c0b"} Oct 31 04:04:18.124915 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:18.124881 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" event={"ID":"c76d973c-8236-442a-9ef4-7c0564a09bd4","Type":"ContainerStarted","Data":"21b7fca64a958c8310ee31638a1dc0821b28c809f1fb8510a6b4bd0b08fb2544"} Oct 31 04:04:19.076080 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.076053 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 04:04:19.136253 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.136221 4426 generic.go:356] "Generic (PLEG): container finished" podID="63c7ff47-3bdc-4426-b49a-f36bb921d757" containerID="06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205" exitCode=0 Oct 31 04:04:19.136668 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.136287 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" Oct 31 04:04:19.136668 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.136314 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" event={"ID":"63c7ff47-3bdc-4426-b49a-f36bb921d757","Type":"ContainerDied","Data":"06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205"} Oct 31 04:04:19.136668 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.136385 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck" event={"ID":"63c7ff47-3bdc-4426-b49a-f36bb921d757","Type":"ContainerDied","Data":"1886dd8085cf519a6f5ed9a7f3c1e5893b768305f13a6ba0b856d845573bebe2"} Oct 31 04:04:19.136668 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.136418 4426 scope.go:122] "RemoveContainer" containerID="06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205" Oct 31 04:04:19.149072 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.149051 4426 scope.go:122] "RemoveContainer" containerID="06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205" Oct 31 04:04:19.149562 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:19.149500 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205\": container with ID starting with 06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205 not found: ID does not exist" containerID="06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205" Oct 31 04:04:19.149562 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.149539 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205"} err="failed to get container status \"06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205\": rpc error: code = NotFound desc = could not find container \"06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205\": container with ID starting with 06c8752eb34eec9e3f4e824f83788fccc8456a34d03669c40a9890cd4b248205 not found: ID does not exist" Oct 31 04:04:19.191537 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.191499 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c7ff47-3bdc-4426-b49a-f36bb921d757-kube-api-access-t2qvl\" (UniqueName: \"kubernetes.io/projected/63c7ff47-3bdc-4426-b49a-f36bb921d757-kube-api-access-t2qvl\") pod \"63c7ff47-3bdc-4426-b49a-f36bb921d757\" (UID: \"63c7ff47-3bdc-4426-b49a-f36bb921d757\") " Oct 31 04:04:19.191733 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.191553 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c7ff47-3bdc-4426-b49a-f36bb921d757-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7ff47-3bdc-4426-b49a-f36bb921d757-plugins-config-volume\") pod \"63c7ff47-3bdc-4426-b49a-f36bb921d757\" (UID: \"63c7ff47-3bdc-4426-b49a-f36bb921d757\") " Oct 31 04:04:19.192387 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.192356 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c7ff47-3bdc-4426-b49a-f36bb921d757-plugins-config-volume" pod "63c7ff47-3bdc-4426-b49a-f36bb921d757" (UID: "63c7ff47-3bdc-4426-b49a-f36bb921d757"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:19.195508 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.195482 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c7ff47-3bdc-4426-b49a-f36bb921d757-kube-api-access-t2qvl" pod "63c7ff47-3bdc-4426-b49a-f36bb921d757" (UID: "63c7ff47-3bdc-4426-b49a-f36bb921d757"). InnerVolumeSpecName "kube-api-access-t2qvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:19.293611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.293576 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qvl\" (UniqueName: \"kubernetes.io/projected/63c7ff47-3bdc-4426-b49a-f36bb921d757-kube-api-access-t2qvl\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:19.293611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.293610 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7ff47-3bdc-4426-b49a-f36bb921d757-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:19.338058 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.338022 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:19.394222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.394189 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76d973c-8236-442a-9ef4-7c0564a09bd4-kube-api-access-lptnc\" (UniqueName: \"kubernetes.io/projected/c76d973c-8236-442a-9ef4-7c0564a09bd4-kube-api-access-lptnc\") pod \"c76d973c-8236-442a-9ef4-7c0564a09bd4\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " Oct 31 04:04:19.394418 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.394255 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-xks-ca\") pod \"c76d973c-8236-442a-9ef4-7c0564a09bd4\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " Oct 31 04:04:19.394418 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.394365 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-openshift-ca\") pod \"c76d973c-8236-442a-9ef4-7c0564a09bd4\" (UID: \"c76d973c-8236-442a-9ef4-7c0564a09bd4\") " Oct 31 04:04:19.394773 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.394739 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-openshift-ca" pod "c76d973c-8236-442a-9ef4-7c0564a09bd4" (UID: "c76d973c-8236-442a-9ef4-7c0564a09bd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:19.394936 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.394909 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:19.396365 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.396343 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-xks-ca" pod "c76d973c-8236-442a-9ef4-7c0564a09bd4" (UID: "c76d973c-8236-442a-9ef4-7c0564a09bd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:04:19.396445 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.396360 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76d973c-8236-442a-9ef4-7c0564a09bd4-kube-api-access-lptnc" pod "c76d973c-8236-442a-9ef4-7c0564a09bd4" (UID: "c76d973c-8236-442a-9ef4-7c0564a09bd4"). InnerVolumeSpecName "kube-api-access-lptnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:19.496865 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.496821 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lptnc\" (UniqueName: \"kubernetes.io/projected/c76d973c-8236-442a-9ef4-7c0564a09bd4-kube-api-access-lptnc\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:19.496865 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.496868 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76d973c-8236-442a-9ef4-7c0564a09bd4-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:19.566002 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.565922 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck"] Oct 31 04:04:19.570728 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:19.570700 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68aa3e-b9d958bc5-c6tck"] Oct 31 04:04:20.149633 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:20.149589 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" event={"ID":"c76d973c-8236-442a-9ef4-7c0564a09bd4","Type":"ContainerDied","Data":"21b7fca64a958c8310ee31638a1dc0821b28c809f1fb8510a6b4bd0b08fb2544"} Oct 31 04:04:20.149633 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:20.149619 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t" Oct 31 04:04:20.150268 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:20.149660 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b7fca64a958c8310ee31638a1dc0821b28c809f1fb8510a6b4bd0b08fb2544" Oct 31 04:04:21.152676 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:21.152641 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c7ff47-3bdc-4426-b49a-f36bb921d757" path="/var/lib/kubelet/pods/63c7ff47-3bdc-4426-b49a-f36bb921d757/volumes" Oct 31 04:04:25.452795 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:25.452758 4426 scope.go:122] "RemoveContainer" containerID="10c57f0bb5db4f5e1820a92b55adca54efdd3161d371a8f8db1d86a914f0ae95" Oct 31 04:04:27.119105 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.119066 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v"] Oct 31 04:04:27.119878 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.119856 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c76d973c-8236-442a-9ef4-7c0564a09bd4" containerName="revoke-keys" Oct 31 04:04:27.119942 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.119882 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76d973c-8236-442a-9ef4-7c0564a09bd4" containerName="revoke-keys" Oct 31 04:04:27.119942 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.119929 4426 cpu_manager.go:414] "containerMap: removing container" podUID="63c7ff47-3bdc-4426-b49a-f36bb921d757" containerName="payload-processing" Oct 31 04:04:27.119942 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.119938 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c7ff47-3bdc-4426-b49a-f36bb921d757" containerName="payload-processing" Oct 31 04:04:27.120116 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.120101 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76d973c-8236-442a-9ef4-7c0564a09bd4" containerName="revoke-keys" Oct 31 04:04:27.120219 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.120140 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c7ff47-3bdc-4426-b49a-f36bb921d757" containerName="payload-processing" Oct 31 04:04:27.125773 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.125744 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:04:27.179480 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.174658 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6e81aea5\"" Oct 31 04:04:27.179480 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.174834 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6e81aea5-dockercfg-jj9d5\"" Oct 31 04:04:27.179480 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.177567 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v"] Oct 31 04:04:27.292200 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.282938 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc"] Oct 31 04:04:27.292200 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.289776 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:04:27.298824 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.298775 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8cpp\" (UniqueName: \"kubernetes.io/projected/0182921c-a48d-47db-a101-5cd137abf755-kube-api-access-v8cpp\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v\" (UID: \"0182921c-a48d-47db-a101-5cd137abf755\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:04:27.298997 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.298935 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0182921c-a48d-47db-a101-5cd137abf755-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v\" (UID: \"0182921c-a48d-47db-a101-5cd137abf755\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:04:27.298997 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.298965 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfdtv\" (UniqueName: \"kubernetes.io/projected/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-kube-api-access-pfdtv\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc\" (UID: \"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:04:27.298997 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.298994 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-plugins-config-volume\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc\" (UID: \"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:04:27.410152 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.404054 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0182921c-a48d-47db-a101-5cd137abf755-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v\" (UID: \"0182921c-a48d-47db-a101-5cd137abf755\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:04:27.410152 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.404110 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfdtv\" (UniqueName: \"kubernetes.io/projected/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-kube-api-access-pfdtv\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc\" (UID: \"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:04:27.410152 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.404166 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-plugins-config-volume\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc\" (UID: \"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:04:27.410152 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.404456 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8cpp\" (UniqueName: \"kubernetes.io/projected/0182921c-a48d-47db-a101-5cd137abf755-kube-api-access-v8cpp\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v\" (UID: \"0182921c-a48d-47db-a101-5cd137abf755\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:04:27.424192 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.419869 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0182921c-a48d-47db-a101-5cd137abf755-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v\" (UID: \"0182921c-a48d-47db-a101-5cd137abf755\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:04:27.424192 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.421022 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-plugins-config-volume\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc\" (UID: \"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:04:27.428721 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.428669 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc"] Oct 31 04:04:27.478925 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.478886 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfdtv\" (UniqueName: \"kubernetes.io/projected/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-kube-api-access-pfdtv\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc\" (UID: \"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:04:27.479783 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.479755 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8cpp\" (UniqueName: \"kubernetes.io/projected/0182921c-a48d-47db-a101-5cd137abf755-kube-api-access-v8cpp\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v\" (UID: \"0182921c-a48d-47db-a101-5cd137abf755\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:04:27.619874 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.619837 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:04:27.741437 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:27.741396 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:04:28.002920 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.002831 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc"] Oct 31 04:04:28.006742 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:04:28.006712 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3583f9e6_d6fd_4d10_bcb8_a259c8fe9a6e.slice/crio-f53724ac96c308e23cd5fd5445cf7817a62bc3c78810284c713a3a650271773e WatchSource:0}: Error finding container f53724ac96c308e23cd5fd5445cf7817a62bc3c78810284c713a3a650271773e: Status 404 returned error can't find the container with id f53724ac96c308e23cd5fd5445cf7817a62bc3c78810284c713a3a650271773e Oct 31 04:04:28.131543 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.131467 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v"] Oct 31 04:04:28.135021 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:04:28.134992 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0182921c_a48d_47db_a101_5cd137abf755.slice/crio-7e02ad3dd521915f0cd5c55bd8d8ab951b989cc6510a591c92acf1c53b30035b WatchSource:0}: Error finding container 7e02ad3dd521915f0cd5c55bd8d8ab951b989cc6510a591c92acf1c53b30035b: Status 404 returned error can't find the container with id 7e02ad3dd521915f0cd5c55bd8d8ab951b989cc6510a591c92acf1c53b30035b Oct 31 04:04:28.338652 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.338548 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" event={"ID":"0182921c-a48d-47db-a101-5cd137abf755","Type":"ContainerStarted","Data":"5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc"} Oct 31 04:04:28.338652 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.338622 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" event={"ID":"0182921c-a48d-47db-a101-5cd137abf755","Type":"ContainerStarted","Data":"7e02ad3dd521915f0cd5c55bd8d8ab951b989cc6510a591c92acf1c53b30035b"} Oct 31 04:04:28.338652 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.338654 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:04:28.340489 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.340452 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" event={"ID":"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e","Type":"ContainerStarted","Data":"1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522"} Oct 31 04:04:28.340640 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.340523 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:04:28.340640 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.340548 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" event={"ID":"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e","Type":"ContainerStarted","Data":"f53724ac96c308e23cd5fd5445cf7817a62bc3c78810284c713a3a650271773e"} Oct 31 04:04:28.450491 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.450424 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" podStartSLOduration=1.450405037 podStartE2EDuration="1.450405037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:28.447060225 +0000 UTC m=+2165.447685450" watchObservedRunningTime="2026-10-31 04:04:28.450405037 +0000 UTC m=+2165.451030267" Oct 31 04:04:28.571097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.571038 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" podStartSLOduration=1.5710228960000001 podStartE2EDuration="1.571022896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:28.557789035 +0000 UTC m=+2165.558414266" watchObservedRunningTime="2026-10-31 04:04:28.571022896 +0000 UTC m=+2165.571648124" Oct 31 04:04:28.722439 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.722405 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft"] Oct 31 04:04:28.722659 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:28.722603 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" containerID="cri-o://2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617" gracePeriod=30 Oct 31 04:04:29.962076 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:29.961998 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7"] Oct 31 04:04:30.062453 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:30.062411 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw"] Oct 31 04:04:32.850297 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:32.850252 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:04:34.008807 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.008739 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 04:04:34.154961 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.154913 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-credential-socket\") pod \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " Oct 31 04:04:34.155205 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.155024 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-podinfo\") pod \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " Oct 31 04:04:34.155205 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.155071 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-token\" (UniqueName: \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-token\") pod \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " Oct 31 04:04:34.155345 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.155245 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-credential-socket" pod "f3efa963-bb10-43b0-a320-8e7d9a38d0df" (UID: "f3efa963-bb10-43b0-a320-8e7d9a38d0df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:34.155345 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.155278 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-data\") pod \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " Oct 31 04:04:34.155345 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.155309 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istiod-ca-cert\") pod \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " Oct 31 04:04:34.155345 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.155340 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-kube-api-access-2sdx4\" (UniqueName: \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-kube-api-access-2sdx4\") pod \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " Oct 31 04:04:34.155566 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.155372 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-socket\") pod \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " Oct 31 04:04:34.155566 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.155407 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-envoy\") pod \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " Oct 31 04:04:34.155566 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.155476 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-certs\") pod \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\" (UID: \"f3efa963-bb10-43b0-a320-8e7d9a38d0df\") " Oct 31 04:04:34.155791 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.155758 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istiod-ca-cert" pod "f3efa963-bb10-43b0-a320-8e7d9a38d0df" (UID: "f3efa963-bb10-43b0-a320-8e7d9a38d0df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:34.156002 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.155965 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-certs" pod "f3efa963-bb10-43b0-a320-8e7d9a38d0df" (UID: "f3efa963-bb10-43b0-a320-8e7d9a38d0df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:34.156606 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.156052 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-socket" pod "f3efa963-bb10-43b0-a320-8e7d9a38d0df" (UID: "f3efa963-bb10-43b0-a320-8e7d9a38d0df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:34.156606 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.156403 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:34.156606 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.156430 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:34.156606 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.156446 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:34.156606 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.156467 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:34.156606 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.156542 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-data" pod "f3efa963-bb10-43b0-a320-8e7d9a38d0df" (UID: "f3efa963-bb10-43b0-a320-8e7d9a38d0df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:34.157824 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.157793 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-kube-api-access-2sdx4" pod "f3efa963-bb10-43b0-a320-8e7d9a38d0df" (UID: "f3efa963-bb10-43b0-a320-8e7d9a38d0df"). InnerVolumeSpecName "kube-api-access-2sdx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:34.157965 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.157929 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-podinfo" pod "f3efa963-bb10-43b0-a320-8e7d9a38d0df" (UID: "f3efa963-bb10-43b0-a320-8e7d9a38d0df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:04:34.158210 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.158183 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-envoy" pod "f3efa963-bb10-43b0-a320-8e7d9a38d0df" (UID: "f3efa963-bb10-43b0-a320-8e7d9a38d0df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:34.158464 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.158440 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-token" pod "f3efa963-bb10-43b0-a320-8e7d9a38d0df" (UID: "f3efa963-bb10-43b0-a320-8e7d9a38d0df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:34.258222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.257932 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:34.258222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.257979 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sdx4\" (UniqueName: \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-kube-api-access-2sdx4\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:34.258222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.257999 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:34.258222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.258014 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:34.258222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.258029 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3efa963-bb10-43b0-a320-8e7d9a38d0df-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:34.435338 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.435290 4426 generic.go:356] "Generic (PLEG): container finished" podID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerID="2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617" exitCode=0 Oct 31 04:04:34.435533 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.435347 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" Oct 31 04:04:34.435533 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.435383 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" event={"ID":"f3efa963-bb10-43b0-a320-8e7d9a38d0df","Type":"ContainerDied","Data":"2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617"} Oct 31 04:04:34.435533 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.435458 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft" event={"ID":"f3efa963-bb10-43b0-a320-8e7d9a38d0df","Type":"ContainerDied","Data":"62c5a040cfd8a7ebcabf935e791636d067d9bb8be2f8d332e4be30b6026be861"} Oct 31 04:04:34.435533 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.435482 4426 scope.go:122] "RemoveContainer" containerID="2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617" Oct 31 04:04:34.450173 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.450111 4426 scope.go:122] "RemoveContainer" containerID="2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617" Oct 31 04:04:34.450512 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:34.450487 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617\": container with ID starting with 2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617 not found: ID does not exist" containerID="2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617" Oct 31 04:04:34.450586 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.450528 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617"} err="failed to get container status \"2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617\": rpc error: code = NotFound desc = could not find container \"2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617\": container with ID starting with 2f43996b4df07737b83f32b756c3cf1c87e400087edd445b20ecba061fb8a617 not found: ID does not exist" Oct 31 04:04:34.593168 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.593121 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft"] Oct 31 04:04:34.597480 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.597460 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-wqdft"] Oct 31 04:04:34.962980 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:34.962934 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" podUID="283c4e57-10b6-46ad-92b6-d490dfff7db2" containerName="payload-processing" containerID="cri-o://af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9" gracePeriod=25 Oct 31 04:04:35.063031 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.062941 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" podUID="26f8a527-8cc0-44f4-a09a-3703e4444154" containerName="payload-pre-processing" containerID="cri-o://14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d" gracePeriod=25 Oct 31 04:04:35.134089 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.134056 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" path="/var/lib/kubelet/pods/f3efa963-bb10-43b0-a320-8e7d9a38d0df/volumes" Oct 31 04:04:35.279195 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.274587 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 04:04:35.368518 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.368491 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 04:04:35.375627 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.375602 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283c4e57-10b6-46ad-92b6-d490dfff7db2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/283c4e57-10b6-46ad-92b6-d490dfff7db2-plugins-config-volume\") pod \"283c4e57-10b6-46ad-92b6-d490dfff7db2\" (UID: \"283c4e57-10b6-46ad-92b6-d490dfff7db2\") " Oct 31 04:04:35.375893 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.375876 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283c4e57-10b6-46ad-92b6-d490dfff7db2-kube-api-access-bgjv8\" (UniqueName: \"kubernetes.io/projected/283c4e57-10b6-46ad-92b6-d490dfff7db2-kube-api-access-bgjv8\") pod \"283c4e57-10b6-46ad-92b6-d490dfff7db2\" (UID: \"283c4e57-10b6-46ad-92b6-d490dfff7db2\") " Oct 31 04:04:35.376006 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.375982 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283c4e57-10b6-46ad-92b6-d490dfff7db2-plugins-config-volume" pod "283c4e57-10b6-46ad-92b6-d490dfff7db2" (UID: "283c4e57-10b6-46ad-92b6-d490dfff7db2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:35.376384 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.376365 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/283c4e57-10b6-46ad-92b6-d490dfff7db2-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:35.377959 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.377939 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283c4e57-10b6-46ad-92b6-d490dfff7db2-kube-api-access-bgjv8" pod "283c4e57-10b6-46ad-92b6-d490dfff7db2" (UID: "283c4e57-10b6-46ad-92b6-d490dfff7db2"). InnerVolumeSpecName "kube-api-access-bgjv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:35.452734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.452698 4426 generic.go:356] "Generic (PLEG): container finished" podID="283c4e57-10b6-46ad-92b6-d490dfff7db2" containerID="af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9" exitCode=0 Oct 31 04:04:35.452937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.452763 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" Oct 31 04:04:35.452937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.452786 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" event={"ID":"283c4e57-10b6-46ad-92b6-d490dfff7db2","Type":"ContainerDied","Data":"af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9"} Oct 31 04:04:35.452937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.452852 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7" event={"ID":"283c4e57-10b6-46ad-92b6-d490dfff7db2","Type":"ContainerDied","Data":"d1022bf2ddc089e68081ff24fe1194e2a0eeb67f82fe052ac261383fca35a9d7"} Oct 31 04:04:35.452937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.452883 4426 scope.go:122] "RemoveContainer" containerID="af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9" Oct 31 04:04:35.454892 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.454687 4426 generic.go:356] "Generic (PLEG): container finished" podID="26f8a527-8cc0-44f4-a09a-3703e4444154" containerID="14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d" exitCode=0 Oct 31 04:04:35.454892 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.454723 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" event={"ID":"26f8a527-8cc0-44f4-a09a-3703e4444154","Type":"ContainerDied","Data":"14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d"} Oct 31 04:04:35.454892 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.454749 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" Oct 31 04:04:35.454892 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.454820 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw" event={"ID":"26f8a527-8cc0-44f4-a09a-3703e4444154","Type":"ContainerDied","Data":"43e3ef5c4035553606126e8e91d802fccce390ed895094018cc4c61fd3c1a928"} Oct 31 04:04:35.465340 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.465317 4426 scope.go:122] "RemoveContainer" containerID="af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9" Oct 31 04:04:35.465627 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:35.465602 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9\": container with ID starting with af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9 not found: ID does not exist" containerID="af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9" Oct 31 04:04:35.465678 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.465636 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9"} err="failed to get container status \"af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9\": rpc error: code = NotFound desc = could not find container \"af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9\": container with ID starting with af81e6b7a93dab104ca6c6bcf36136c50f5508bf1f3d6cb49322595559f8b9d9 not found: ID does not exist" Oct 31 04:04:35.465678 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.465654 4426 scope.go:122] "RemoveContainer" containerID="14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d" Oct 31 04:04:35.475730 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.475713 4426 scope.go:122] "RemoveContainer" containerID="14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d" Oct 31 04:04:35.476013 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:35.475995 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d\": container with ID starting with 14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d not found: ID does not exist" containerID="14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d" Oct 31 04:04:35.476058 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.476022 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d"} err="failed to get container status \"14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d\": rpc error: code = NotFound desc = could not find container \"14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d\": container with ID starting with 14574fc715e8efd1a0a0dff17fa95a5232b05afbeb2158e633c198a143fb3f0d not found: ID does not exist" Oct 31 04:04:35.477394 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.477378 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f8a527-8cc0-44f4-a09a-3703e4444154-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f8a527-8cc0-44f4-a09a-3703e4444154-plugins-config-volume\") pod \"26f8a527-8cc0-44f4-a09a-3703e4444154\" (UID: \"26f8a527-8cc0-44f4-a09a-3703e4444154\") " Oct 31 04:04:35.477586 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.477574 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f8a527-8cc0-44f4-a09a-3703e4444154-kube-api-access-42vv4\" (UniqueName: \"kubernetes.io/projected/26f8a527-8cc0-44f4-a09a-3703e4444154-kube-api-access-42vv4\") pod \"26f8a527-8cc0-44f4-a09a-3703e4444154\" (UID: \"26f8a527-8cc0-44f4-a09a-3703e4444154\") " Oct 31 04:04:35.477725 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.477700 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f8a527-8cc0-44f4-a09a-3703e4444154-plugins-config-volume" pod "26f8a527-8cc0-44f4-a09a-3703e4444154" (UID: "26f8a527-8cc0-44f4-a09a-3703e4444154"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:35.477993 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.477980 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f8a527-8cc0-44f4-a09a-3703e4444154-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:35.478061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.477997 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgjv8\" (UniqueName: \"kubernetes.io/projected/283c4e57-10b6-46ad-92b6-d490dfff7db2-kube-api-access-bgjv8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:35.479536 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.479516 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f8a527-8cc0-44f4-a09a-3703e4444154-kube-api-access-42vv4" pod "26f8a527-8cc0-44f4-a09a-3703e4444154" (UID: "26f8a527-8cc0-44f4-a09a-3703e4444154"). InnerVolumeSpecName "kube-api-access-42vv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:35.579723 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.579644 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42vv4\" (UniqueName: \"kubernetes.io/projected/26f8a527-8cc0-44f4-a09a-3703e4444154-kube-api-access-42vv4\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:35.605894 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.605859 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7"] Oct 31 04:04:35.610311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.610284 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-976cea-c5f6c45d9-h5bq7"] Oct 31 04:04:35.871483 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.871387 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw"] Oct 31 04:04:35.875887 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:35.875856 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-976cea-7bf98cc5d7-zpmhw"] Oct 31 04:04:37.092559 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:37.092522 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp"] Oct 31 04:04:37.092953 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:37.092744 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerName="istio-proxy" containerID="cri-o://0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db" gracePeriod=30 Oct 31 04:04:37.137758 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:37.137721 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f8a527-8cc0-44f4-a09a-3703e4444154" path="/var/lib/kubelet/pods/26f8a527-8cc0-44f4-a09a-3703e4444154/volumes" Oct 31 04:04:37.138266 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:37.138248 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283c4e57-10b6-46ad-92b6-d490dfff7db2" path="/var/lib/kubelet/pods/283c4e57-10b6-46ad-92b6-d490dfff7db2/volumes" Oct 31 04:04:37.786753 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:37.786714 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:04:39.363152 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:39.362603 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:04:39.366146 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:39.364031 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:04:40.200141 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.200089 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r"] Oct 31 04:04:40.201257 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.201232 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" Oct 31 04:04:40.201323 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.201261 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" Oct 31 04:04:40.201361 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.201329 4426 cpu_manager.go:414] "containerMap: removing container" podUID="283c4e57-10b6-46ad-92b6-d490dfff7db2" containerName="payload-processing" Oct 31 04:04:40.201361 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.201339 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283c4e57-10b6-46ad-92b6-d490dfff7db2" containerName="payload-processing" Oct 31 04:04:40.201361 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.201357 4426 cpu_manager.go:414] "containerMap: removing container" podUID="26f8a527-8cc0-44f4-a09a-3703e4444154" containerName="payload-pre-processing" Oct 31 04:04:40.201454 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.201367 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f8a527-8cc0-44f4-a09a-3703e4444154" containerName="payload-pre-processing" Oct 31 04:04:40.201553 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.201539 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="283c4e57-10b6-46ad-92b6-d490dfff7db2" containerName="payload-processing" Oct 31 04:04:40.201588 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.201562 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f8a527-8cc0-44f4-a09a-3703e4444154" containerName="payload-pre-processing" Oct 31 04:04:40.201588 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.201578 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3efa963-bb10-43b0-a320-8e7d9a38d0df" containerName="istio-proxy" Oct 31 04:04:40.207808 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.207774 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:40.256955 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.256920 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6"] Oct 31 04:04:40.268543 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.268489 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r"] Oct 31 04:04:40.269063 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.268966 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:40.288182 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.288076 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec27414d-fk68r\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:40.288365 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.288237 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf98297-xrdz6\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:40.288365 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.288293 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec27414d-fk68r\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:40.288506 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.288462 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7crp\" (UniqueName: \"kubernetes.io/projected/50c91447-1f10-42d6-8702-377d3e5f1115-kube-api-access-r7crp\") pod \"maas-api-revoke-sub-ec27414d-fk68r\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:40.288571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.288504 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k69f\" (UniqueName: \"kubernetes.io/projected/5f947877-2492-415b-814b-fca6c1ee9bb8-kube-api-access-4k69f\") pod \"maas-api-revoke-sub-3bf98297-xrdz6\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:40.288571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.288531 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf98297-xrdz6\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:40.293604 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.293569 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:04:40.293829 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.293699 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:04:40.293829 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.293804 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:04:40.294038 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.294014 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:04:40.314026 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.313985 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6"] Oct 31 04:04:40.389532 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.389491 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec27414d-fk68r\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:40.390078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.389589 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf98297-xrdz6\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:40.390078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.389634 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec27414d-fk68r\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:40.390078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.389763 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7crp\" (UniqueName: \"kubernetes.io/projected/50c91447-1f10-42d6-8702-377d3e5f1115-kube-api-access-r7crp\") pod \"maas-api-revoke-sub-ec27414d-fk68r\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:40.390078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.389795 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k69f\" (UniqueName: \"kubernetes.io/projected/5f947877-2492-415b-814b-fca6c1ee9bb8-kube-api-access-4k69f\") pod \"maas-api-revoke-sub-3bf98297-xrdz6\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:40.390078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.389820 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf98297-xrdz6\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:40.391375 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.390974 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec27414d-fk68r\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:40.391375 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.391320 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf98297-xrdz6\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:40.393098 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.393071 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec27414d-fk68r\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:40.394055 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.394031 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf98297-xrdz6\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:40.448655 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.448618 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k69f\" (UniqueName: \"kubernetes.io/projected/5f947877-2492-415b-814b-fca6c1ee9bb8-kube-api-access-4k69f\") pod \"maas-api-revoke-sub-3bf98297-xrdz6\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:40.450832 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.450743 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7crp\" (UniqueName: \"kubernetes.io/projected/50c91447-1f10-42d6-8702-377d3e5f1115-kube-api-access-r7crp\") pod \"maas-api-revoke-sub-ec27414d-fk68r\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:40.535247 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.535203 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:40.597537 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:40.597240 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:41.515856 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:41.515809 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r"] Oct 31 04:04:41.532313 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:04:41.532250 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c91447_1f10_42d6_8702_377d3e5f1115.slice/crio-52ec04e89de10e9c52b21faf48718628f1893085a2858d7522900b8dd1c75c9c WatchSource:0}: Error finding container 52ec04e89de10e9c52b21faf48718628f1893085a2858d7522900b8dd1c75c9c: Status 404 returned error can't find the container with id 52ec04e89de10e9c52b21faf48718628f1893085a2858d7522900b8dd1c75c9c Oct 31 04:04:41.711466 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:41.711277 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" event={"ID":"50c91447-1f10-42d6-8702-377d3e5f1115","Type":"ContainerStarted","Data":"52ec04e89de10e9c52b21faf48718628f1893085a2858d7522900b8dd1c75c9c"} Oct 31 04:04:41.889420 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:41.888357 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6"] Oct 31 04:04:42.630798 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.630732 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 04:04:42.722415 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.722375 4426 generic.go:356] "Generic (PLEG): container finished" podID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerID="0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db" exitCode=0 Oct 31 04:04:42.722611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.722411 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" event={"ID":"7f0a72fe-ce05-498d-8e57-bb41826904a0","Type":"ContainerDied","Data":"0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db"} Oct 31 04:04:42.722611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.722469 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" event={"ID":"7f0a72fe-ce05-498d-8e57-bb41826904a0","Type":"ContainerDied","Data":"59ab2338d8176dc9be7d3472dabce2cb661846778973686b571429c9e155e983"} Oct 31 04:04:42.722611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.722492 4426 scope.go:122] "RemoveContainer" containerID="0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db" Oct 31 04:04:42.722611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.722429 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp" Oct 31 04:04:42.724618 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.724591 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" event={"ID":"50c91447-1f10-42d6-8702-377d3e5f1115","Type":"ContainerStarted","Data":"c4859941939aaf5b08782e192a8d47055aea494807b066e6447f6331ff468d1c"} Oct 31 04:04:42.726689 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.726665 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" event={"ID":"5f947877-2492-415b-814b-fca6c1ee9bb8","Type":"ContainerStarted","Data":"0b9253d95096019e99e4a9ccdb67c57f96ad94f13347d910d06f2d585d0c4e1d"} Oct 31 04:04:42.726823 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.726703 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" event={"ID":"5f947877-2492-415b-814b-fca6c1ee9bb8","Type":"ContainerStarted","Data":"14c66aeb6c716792ee8b128f724368e5479e65dda0abd577e5651d5ecb984f82"} Oct 31 04:04:42.734346 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.734318 4426 scope.go:122] "RemoveContainer" containerID="0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db" Oct 31 04:04:42.734662 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:04:42.734638 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db\": container with ID starting with 0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db not found: ID does not exist" containerID="0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db" Oct 31 04:04:42.734752 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.734670 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db"} err="failed to get container status \"0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db\": rpc error: code = NotFound desc = could not find container \"0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db\": container with ID starting with 0d7ce98f1ee1b396b3f320a4000a00f0161b6ba1f0bbbe5ac02bff934dd3f5db not found: ID does not exist" Oct 31 04:04:42.744138 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744099 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-credential-socket\") pod \"7f0a72fe-ce05-498d-8e57-bb41826904a0\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " Oct 31 04:04:42.744343 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744242 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-envoy\") pod \"7f0a72fe-ce05-498d-8e57-bb41826904a0\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " Oct 31 04:04:42.744417 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744344 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-token\" (UniqueName: \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-token\") pod \"7f0a72fe-ce05-498d-8e57-bb41826904a0\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " Oct 31 04:04:42.744417 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744392 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-credential-socket" pod "7f0a72fe-ce05-498d-8e57-bb41826904a0" (UID: "7f0a72fe-ce05-498d-8e57-bb41826904a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:42.744532 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744449 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-kube-api-access-r4t4h\" (UniqueName: \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-kube-api-access-r4t4h\") pod \"7f0a72fe-ce05-498d-8e57-bb41826904a0\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " Oct 31 04:04:42.744532 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744490 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-podinfo\") pod \"7f0a72fe-ce05-498d-8e57-bb41826904a0\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " Oct 31 04:04:42.744532 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744512 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-socket\") pod \"7f0a72fe-ce05-498d-8e57-bb41826904a0\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " Oct 31 04:04:42.744701 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744555 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-data\") pod \"7f0a72fe-ce05-498d-8e57-bb41826904a0\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " Oct 31 04:04:42.744701 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744580 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0a72fe-ce05-498d-8e57-bb41826904a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0a72fe-ce05-498d-8e57-bb41826904a0-istiod-ca-cert\") pod \"7f0a72fe-ce05-498d-8e57-bb41826904a0\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " Oct 31 04:04:42.744701 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744646 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-certs\") pod \"7f0a72fe-ce05-498d-8e57-bb41826904a0\" (UID: \"7f0a72fe-ce05-498d-8e57-bb41826904a0\") " Oct 31 04:04:42.744869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744818 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-socket" pod "7f0a72fe-ce05-498d-8e57-bb41826904a0" (UID: "7f0a72fe-ce05-498d-8e57-bb41826904a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:42.744869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.744841 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-certs" pod "7f0a72fe-ce05-498d-8e57-bb41826904a0" (UID: "7f0a72fe-ce05-498d-8e57-bb41826904a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:42.745078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.745052 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0a72fe-ce05-498d-8e57-bb41826904a0-istiod-ca-cert" pod "7f0a72fe-ce05-498d-8e57-bb41826904a0" (UID: "7f0a72fe-ce05-498d-8e57-bb41826904a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:42.745531 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.745508 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:42.745691 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.745538 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:42.745691 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.745555 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0a72fe-ce05-498d-8e57-bb41826904a0-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:42.745691 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.745569 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:42.745869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.745744 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-data" pod "7f0a72fe-ce05-498d-8e57-bb41826904a0" (UID: "7f0a72fe-ce05-498d-8e57-bb41826904a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:42.747192 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.747164 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-kube-api-access-r4t4h" pod "7f0a72fe-ce05-498d-8e57-bb41826904a0" (UID: "7f0a72fe-ce05-498d-8e57-bb41826904a0"). InnerVolumeSpecName "kube-api-access-r4t4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:42.747293 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.747199 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-envoy" pod "7f0a72fe-ce05-498d-8e57-bb41826904a0" (UID: "7f0a72fe-ce05-498d-8e57-bb41826904a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:42.747293 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.747272 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-token" pod "7f0a72fe-ce05-498d-8e57-bb41826904a0" (UID: "7f0a72fe-ce05-498d-8e57-bb41826904a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:42.750447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.750424 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-podinfo" pod "7f0a72fe-ce05-498d-8e57-bb41826904a0" (UID: "7f0a72fe-ce05-498d-8e57-bb41826904a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:04:42.847114 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.847077 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:42.847114 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.847109 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:42.847385 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.847121 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4t4h\" (UniqueName: \"kubernetes.io/projected/7f0a72fe-ce05-498d-8e57-bb41826904a0-kube-api-access-r4t4h\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:42.847385 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.847150 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:42.847385 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.847159 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0a72fe-ce05-498d-8e57-bb41826904a0-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:42.900707 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:42.900647 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" podStartSLOduration=2.900632352 podStartE2EDuration="2.900632352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:42.898599198 +0000 UTC m=+2179.899224438" watchObservedRunningTime="2026-10-31 04:04:42.900632352 +0000 UTC m=+2179.901257579" Oct 31 04:04:43.058408 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:43.058350 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" podStartSLOduration=3.058336715 podStartE2EDuration="3.058336715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:43.055534121 +0000 UTC m=+2180.056159348" watchObservedRunningTime="2026-10-31 04:04:43.058336715 +0000 UTC m=+2180.058961943" Oct 31 04:04:43.163791 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:43.163694 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp"] Oct 31 04:04:43.168892 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:43.168858 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-pk4lp"] Oct 31 04:04:43.738113 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:43.738074 4426 generic.go:356] "Generic (PLEG): container finished" podID="50c91447-1f10-42d6-8702-377d3e5f1115" containerID="c4859941939aaf5b08782e192a8d47055aea494807b066e6447f6331ff468d1c" exitCode=0 Oct 31 04:04:43.738611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:43.738164 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" event={"ID":"50c91447-1f10-42d6-8702-377d3e5f1115","Type":"ContainerDied","Data":"c4859941939aaf5b08782e192a8d47055aea494807b066e6447f6331ff468d1c"} Oct 31 04:04:43.739998 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:43.739968 4426 generic.go:356] "Generic (PLEG): container finished" podID="5f947877-2492-415b-814b-fca6c1ee9bb8" containerID="0b9253d95096019e99e4a9ccdb67c57f96ad94f13347d910d06f2d585d0c4e1d" exitCode=0 Oct 31 04:04:43.740149 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:43.740015 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" event={"ID":"5f947877-2492-415b-814b-fca6c1ee9bb8","Type":"ContainerDied","Data":"0b9253d95096019e99e4a9ccdb67c57f96ad94f13347d910d06f2d585d0c4e1d"} Oct 31 04:04:44.948060 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:44.948033 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:44.982043 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:44.982011 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:45.078580 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.078488 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-xks-ca\") pod \"50c91447-1f10-42d6-8702-377d3e5f1115\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " Oct 31 04:04:45.078756 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.078620 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-openshift-ca\") pod \"50c91447-1f10-42d6-8702-377d3e5f1115\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " Oct 31 04:04:45.078756 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.078660 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-openshift-ca\") pod \"5f947877-2492-415b-814b-fca6c1ee9bb8\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " Oct 31 04:04:45.078756 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.078728 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f947877-2492-415b-814b-fca6c1ee9bb8-kube-api-access-4k69f\" (UniqueName: \"kubernetes.io/projected/5f947877-2492-415b-814b-fca6c1ee9bb8-kube-api-access-4k69f\") pod \"5f947877-2492-415b-814b-fca6c1ee9bb8\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " Oct 31 04:04:45.078919 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.078777 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c91447-1f10-42d6-8702-377d3e5f1115-kube-api-access-r7crp\" (UniqueName: \"kubernetes.io/projected/50c91447-1f10-42d6-8702-377d3e5f1115-kube-api-access-r7crp\") pod \"50c91447-1f10-42d6-8702-377d3e5f1115\" (UID: \"50c91447-1f10-42d6-8702-377d3e5f1115\") " Oct 31 04:04:45.078975 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.078925 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-xks-ca\") pod \"5f947877-2492-415b-814b-fca6c1ee9bb8\" (UID: \"5f947877-2492-415b-814b-fca6c1ee9bb8\") " Oct 31 04:04:45.079101 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.079069 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-openshift-ca" pod "50c91447-1f10-42d6-8702-377d3e5f1115" (UID: "50c91447-1f10-42d6-8702-377d3e5f1115"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:45.079194 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.079107 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-openshift-ca" pod "5f947877-2492-415b-814b-fca6c1ee9bb8" (UID: "5f947877-2492-415b-814b-fca6c1ee9bb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:45.079506 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.079479 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:45.079596 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.079511 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:45.082186 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.081267 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c91447-1f10-42d6-8702-377d3e5f1115-kube-api-access-r7crp" pod "50c91447-1f10-42d6-8702-377d3e5f1115" (UID: "50c91447-1f10-42d6-8702-377d3e5f1115"). InnerVolumeSpecName "kube-api-access-r7crp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:45.082186 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.081298 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-xks-ca" pod "5f947877-2492-415b-814b-fca6c1ee9bb8" (UID: "5f947877-2492-415b-814b-fca6c1ee9bb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:04:45.082186 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.081420 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-xks-ca" pod "50c91447-1f10-42d6-8702-377d3e5f1115" (UID: "50c91447-1f10-42d6-8702-377d3e5f1115"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:04:45.082186 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.081651 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f947877-2492-415b-814b-fca6c1ee9bb8-kube-api-access-4k69f" pod "5f947877-2492-415b-814b-fca6c1ee9bb8" (UID: "5f947877-2492-415b-814b-fca6c1ee9bb8"). InnerVolumeSpecName "kube-api-access-4k69f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:45.132807 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.132770 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" path="/var/lib/kubelet/pods/7f0a72fe-ce05-498d-8e57-bb41826904a0/volumes" Oct 31 04:04:45.180680 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.180647 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7crp\" (UniqueName: \"kubernetes.io/projected/50c91447-1f10-42d6-8702-377d3e5f1115-kube-api-access-r7crp\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:45.180680 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.180676 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f947877-2492-415b-814b-fca6c1ee9bb8-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:45.180680 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.180687 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c91447-1f10-42d6-8702-377d3e5f1115-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:45.181009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.180696 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k69f\" (UniqueName: \"kubernetes.io/projected/5f947877-2492-415b-814b-fca6c1ee9bb8-kube-api-access-4k69f\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:45.766150 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.765202 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" Oct 31 04:04:45.766150 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.765243 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r" event={"ID":"50c91447-1f10-42d6-8702-377d3e5f1115","Type":"ContainerDied","Data":"52ec04e89de10e9c52b21faf48718628f1893085a2858d7522900b8dd1c75c9c"} Oct 31 04:04:45.766150 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.765311 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52ec04e89de10e9c52b21faf48718628f1893085a2858d7522900b8dd1c75c9c" Oct 31 04:04:45.768997 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.768961 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" event={"ID":"5f947877-2492-415b-814b-fca6c1ee9bb8","Type":"ContainerDied","Data":"14c66aeb6c716792ee8b128f724368e5479e65dda0abd577e5651d5ecb984f82"} Oct 31 04:04:45.769230 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.769018 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14c66aeb6c716792ee8b128f724368e5479e65dda0abd577e5651d5ecb984f82" Oct 31 04:04:45.769230 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:45.769033 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6" Oct 31 04:04:47.220458 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:47.220422 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452"] Oct 31 04:04:47.249698 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:47.249662 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4a3a5-9f452"] Oct 31 04:04:48.456338 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:48.456289 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t"] Oct 31 04:04:48.502096 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:48.502057 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-976cea-jm28t"] Oct 31 04:04:49.133151 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:49.133085 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5064d4e6-b80b-4792-8fce-5284d5369af1" path="/var/lib/kubelet/pods/5064d4e6-b80b-4792-8fce-5284d5369af1/volumes" Oct 31 04:04:49.133872 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:49.133850 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76d973c-8236-442a-9ef4-7c0564a09bd4" path="/var/lib/kubelet/pods/c76d973c-8236-442a-9ef4-7c0564a09bd4/volumes" Oct 31 04:04:53.640704 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.640659 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg"] Oct 31 04:04:53.641814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.641788 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerName="istio-proxy" Oct 31 04:04:53.641895 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.641821 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerName="istio-proxy" Oct 31 04:04:53.641936 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.641915 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5f947877-2492-415b-814b-fca6c1ee9bb8" containerName="revoke-keys" Oct 31 04:04:53.641936 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.641924 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f947877-2492-415b-814b-fca6c1ee9bb8" containerName="revoke-keys" Oct 31 04:04:53.641999 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.641941 4426 cpu_manager.go:414] "containerMap: removing container" podUID="50c91447-1f10-42d6-8702-377d3e5f1115" containerName="revoke-keys" Oct 31 04:04:53.641999 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.641951 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c91447-1f10-42d6-8702-377d3e5f1115" containerName="revoke-keys" Oct 31 04:04:53.642172 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.642151 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f947877-2492-415b-814b-fca6c1ee9bb8" containerName="revoke-keys" Oct 31 04:04:53.642280 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.642180 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c91447-1f10-42d6-8702-377d3e5f1115" containerName="revoke-keys" Oct 31 04:04:53.642280 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.642199 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0a72fe-ce05-498d-8e57-bb41826904a0" containerName="istio-proxy" Oct 31 04:04:53.647653 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.647622 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:04:53.703036 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.702995 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg"] Oct 31 04:04:53.740946 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.740911 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3473a4\"" Oct 31 04:04:53.741229 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.741092 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3473a4-dockercfg-wx5hf\"" Oct 31 04:04:53.845015 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.844969 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf44b\" (UniqueName: \"kubernetes.io/projected/e9d7dd6d-4880-4dc2-b785-93c517a1d176-kube-api-access-rf44b\") pod \"payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg\" (UID: \"e9d7dd6d-4880-4dc2-b785-93c517a1d176\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:04:53.845691 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.845662 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d7dd6d-4880-4dc2-b785-93c517a1d176-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg\" (UID: \"e9d7dd6d-4880-4dc2-b785-93c517a1d176\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:04:53.881904 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.881866 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb"] Oct 31 04:04:53.892686 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.892607 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:04:53.936979 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.936935 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb"] Oct 31 04:04:53.950376 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.950161 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d7dd6d-4880-4dc2-b785-93c517a1d176-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg\" (UID: \"e9d7dd6d-4880-4dc2-b785-93c517a1d176\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:04:53.950376 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.950320 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf44b\" (UniqueName: \"kubernetes.io/projected/e9d7dd6d-4880-4dc2-b785-93c517a1d176-kube-api-access-rf44b\") pod \"payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg\" (UID: \"e9d7dd6d-4880-4dc2-b785-93c517a1d176\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:04:53.951038 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:53.951013 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d7dd6d-4880-4dc2-b785-93c517a1d176-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg\" (UID: \"e9d7dd6d-4880-4dc2-b785-93c517a1d176\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:04:54.013617 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.013580 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf44b\" (UniqueName: \"kubernetes.io/projected/e9d7dd6d-4880-4dc2-b785-93c517a1d176-kube-api-access-rf44b\") pod \"payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg\" (UID: \"e9d7dd6d-4880-4dc2-b785-93c517a1d176\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:04:54.052159 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.052103 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqnq4\" (UniqueName: \"kubernetes.io/projected/2dd1b4c2-d57e-407e-805c-85158275e778-kube-api-access-rqnq4\") pod \"payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb\" (UID: \"2dd1b4c2-d57e-407e-805c-85158275e778\") " pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:04:54.052353 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.052309 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd1b4c2-d57e-407e-805c-85158275e778-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb\" (UID: \"2dd1b4c2-d57e-407e-805c-85158275e778\") " pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:04:54.154365 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.154274 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd1b4c2-d57e-407e-805c-85158275e778-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb\" (UID: \"2dd1b4c2-d57e-407e-805c-85158275e778\") " pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:04:54.154524 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.154428 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnq4\" (UniqueName: \"kubernetes.io/projected/2dd1b4c2-d57e-407e-805c-85158275e778-kube-api-access-rqnq4\") pod \"payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb\" (UID: \"2dd1b4c2-d57e-407e-805c-85158275e778\") " pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:04:54.154999 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.154976 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd1b4c2-d57e-407e-805c-85158275e778-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb\" (UID: \"2dd1b4c2-d57e-407e-805c-85158275e778\") " pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:04:54.239151 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.231233 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnq4\" (UniqueName: \"kubernetes.io/projected/2dd1b4c2-d57e-407e-805c-85158275e778-kube-api-access-rqnq4\") pod \"payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb\" (UID: \"2dd1b4c2-d57e-407e-805c-85158275e778\") " pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:04:54.266976 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.266519 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:04:54.511477 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.511439 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:04:54.706210 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.701656 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg"] Oct 31 04:04:54.859006 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.858968 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" event={"ID":"e9d7dd6d-4880-4dc2-b785-93c517a1d176","Type":"ContainerStarted","Data":"6041b4efa1fbbc1a896f89db46cc58953941a11d229a81bc41d1a5cb635973e5"} Oct 31 04:04:54.902761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:54.902734 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb"] Oct 31 04:04:54.906136 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:04:54.906095 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd1b4c2_d57e_407e_805c_85158275e778.slice/crio-c7f11be0b43f037cd487e7d5c548de2565d53ce29dbfd8c220788b3f0767581d WatchSource:0}: Error finding container c7f11be0b43f037cd487e7d5c548de2565d53ce29dbfd8c220788b3f0767581d: Status 404 returned error can't find the container with id c7f11be0b43f037cd487e7d5c548de2565d53ce29dbfd8c220788b3f0767581d Oct 31 04:04:55.266534 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.266435 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5"] Oct 31 04:04:55.271562 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.271536 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:55.326824 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.326790 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5"] Oct 31 04:04:55.328818 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.328794 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:04:55.330372 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.330347 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:04:55.394871 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.394833 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:55.395050 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.394950 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:55.395050 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.395020 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gw5p\" (UniqueName: \"kubernetes.io/projected/595ba575-a34a-4a82-a76d-4c6b20e06bfd-kube-api-access-4gw5p\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:55.419942 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.419912 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:04:55.420113 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.419958 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:04:55.496877 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.496841 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:55.497055 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.496921 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:55.497055 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.496968 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gw5p\" (UniqueName: \"kubernetes.io/projected/595ba575-a34a-4a82-a76d-4c6b20e06bfd-kube-api-access-4gw5p\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:55.497656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.497629 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:55.499372 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.499347 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:55.557754 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.557675 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gw5p\" (UniqueName: \"kubernetes.io/projected/595ba575-a34a-4a82-a76d-4c6b20e06bfd-kube-api-access-4gw5p\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:55.587330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.587297 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:55.870057 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.869945 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" event={"ID":"2dd1b4c2-d57e-407e-805c-85158275e778","Type":"ContainerStarted","Data":"15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480"} Oct 31 04:04:55.870057 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.870034 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:04:55.870592 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.870061 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" event={"ID":"2dd1b4c2-d57e-407e-805c-85158275e778","Type":"ContainerStarted","Data":"c7f11be0b43f037cd487e7d5c548de2565d53ce29dbfd8c220788b3f0767581d"} Oct 31 04:04:55.872102 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.872044 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" event={"ID":"e9d7dd6d-4880-4dc2-b785-93c517a1d176","Type":"ContainerStarted","Data":"2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315"} Oct 31 04:04:55.872254 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.872152 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:04:55.982238 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:55.982182 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" podStartSLOduration=2.982164424 podStartE2EDuration="2.982164424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:55.978707106 +0000 UTC m=+2192.979332333" watchObservedRunningTime="2026-10-31 04:04:55.982164424 +0000 UTC m=+2192.982789652" Oct 31 04:04:56.002449 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:56.002417 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5"] Oct 31 04:04:56.006045 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:04:56.006010 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595ba575_a34a_4a82_a76d_4c6b20e06bfd.slice/crio-bca640677ac0bfa3c85f596ccbabeab8f4c7e331d522553074f4d7ea1877e47c WatchSource:0}: Error finding container bca640677ac0bfa3c85f596ccbabeab8f4c7e331d522553074f4d7ea1877e47c: Status 404 returned error can't find the container with id bca640677ac0bfa3c85f596ccbabeab8f4c7e331d522553074f4d7ea1877e47c Oct 31 04:04:56.090113 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:56.090058 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" podStartSLOduration=3.090043039 podStartE2EDuration="3.090043039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:56.087108863 +0000 UTC m=+2193.087734090" watchObservedRunningTime="2026-10-31 04:04:56.090043039 +0000 UTC m=+2193.090668267" Oct 31 04:04:56.885571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:56.885532 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" event={"ID":"595ba575-a34a-4a82-a76d-4c6b20e06bfd","Type":"ContainerStarted","Data":"fc05b1d6b9fa56b0c303274db1b8da7963ba3e48876f79745aa86c8539dc71ff"} Oct 31 04:04:56.886021 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:56.885591 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" event={"ID":"595ba575-a34a-4a82-a76d-4c6b20e06bfd","Type":"ContainerStarted","Data":"bca640677ac0bfa3c85f596ccbabeab8f4c7e331d522553074f4d7ea1877e47c"} Oct 31 04:04:56.999282 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:56.999222 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" podStartSLOduration=1.999207194 podStartE2EDuration="1.999207194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:56.993687625 +0000 UTC m=+2193.994312854" watchObservedRunningTime="2026-10-31 04:04:56.999207194 +0000 UTC m=+2193.999832421" Oct 31 04:04:57.895723 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:57.895688 4426 generic.go:356] "Generic (PLEG): container finished" podID="595ba575-a34a-4a82-a76d-4c6b20e06bfd" containerID="fc05b1d6b9fa56b0c303274db1b8da7963ba3e48876f79745aa86c8539dc71ff" exitCode=0 Oct 31 04:04:57.896148 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:57.895783 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" event={"ID":"595ba575-a34a-4a82-a76d-4c6b20e06bfd","Type":"ContainerDied","Data":"fc05b1d6b9fa56b0c303274db1b8da7963ba3e48876f79745aa86c8539dc71ff"} Oct 31 04:04:59.076372 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.076339 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:04:59.253821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.253727 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-xks-ca\") pod \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " Oct 31 04:04:59.254027 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.254005 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595ba575-a34a-4a82-a76d-4c6b20e06bfd-kube-api-access-4gw5p\" (UniqueName: \"kubernetes.io/projected/595ba575-a34a-4a82-a76d-4c6b20e06bfd-kube-api-access-4gw5p\") pod \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " Oct 31 04:04:59.254103 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.254088 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-openshift-ca\") pod \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\" (UID: \"595ba575-a34a-4a82-a76d-4c6b20e06bfd\") " Oct 31 04:04:59.254446 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.254414 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-openshift-ca" pod "595ba575-a34a-4a82-a76d-4c6b20e06bfd" (UID: "595ba575-a34a-4a82-a76d-4c6b20e06bfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:59.254932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.254907 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:59.256031 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.256010 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-xks-ca" pod "595ba575-a34a-4a82-a76d-4c6b20e06bfd" (UID: "595ba575-a34a-4a82-a76d-4c6b20e06bfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:04:59.256395 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.256373 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595ba575-a34a-4a82-a76d-4c6b20e06bfd-kube-api-access-4gw5p" pod "595ba575-a34a-4a82-a76d-4c6b20e06bfd" (UID: "595ba575-a34a-4a82-a76d-4c6b20e06bfd"). InnerVolumeSpecName "kube-api-access-4gw5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:59.356729 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.356690 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gw5p\" (UniqueName: \"kubernetes.io/projected/595ba575-a34a-4a82-a76d-4c6b20e06bfd-kube-api-access-4gw5p\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:59.356729 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.356721 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595ba575-a34a-4a82-a76d-4c6b20e06bfd-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:04:59.918540 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.918491 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" event={"ID":"595ba575-a34a-4a82-a76d-4c6b20e06bfd","Type":"ContainerDied","Data":"bca640677ac0bfa3c85f596ccbabeab8f4c7e331d522553074f4d7ea1877e47c"} Oct 31 04:04:59.918735 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.918615 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bca640677ac0bfa3c85f596ccbabeab8f4c7e331d522553074f4d7ea1877e47c" Oct 31 04:04:59.918735 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:04:59.918703 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5" Oct 31 04:05:02.864195 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:05:02.860043 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:05:06.785981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.785898 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb"] Oct 31 04:05:06.786893 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.786872 4426 cpu_manager.go:414] "containerMap: removing container" podUID="595ba575-a34a-4a82-a76d-4c6b20e06bfd" containerName="revoke-keys" Oct 31 04:05:06.786950 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.786895 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595ba575-a34a-4a82-a76d-4c6b20e06bfd" containerName="revoke-keys" Oct 31 04:05:06.787048 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.787039 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="595ba575-a34a-4a82-a76d-4c6b20e06bfd" containerName="revoke-keys" Oct 31 04:05:06.792251 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.792225 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:06.848352 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.848315 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb"] Oct 31 04:05:06.875476 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.875442 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:06.875667 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.875652 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:06.875719 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.875684 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2wh\" (UniqueName: \"kubernetes.io/projected/925cd563-0f7b-439d-a607-7acd7326c922-kube-api-access-9t2wh\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:06.887687 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.887364 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:05:06.887687 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.887399 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:05:06.887687 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.887533 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:05:06.887687 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.887609 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:05:06.888683 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.888662 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:05:06.888913 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.888893 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:05:06.977340 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.977302 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:06.977574 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.977349 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2wh\" (UniqueName: \"kubernetes.io/projected/925cd563-0f7b-439d-a607-7acd7326c922-kube-api-access-9t2wh\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:06.977574 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.977477 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:06.978075 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.978053 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:06.980018 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:06.979994 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:07.054446 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:07.042980 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2wh\" (UniqueName: \"kubernetes.io/projected/925cd563-0f7b-439d-a607-7acd7326c922-kube-api-access-9t2wh\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:07.106724 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:07.106688 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:07.529902 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:07.529867 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb"] Oct 31 04:05:07.534987 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:05:07.534959 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925cd563_0f7b_439d_a607_7acd7326c922.slice/crio-fcb391e7dd2319314e28dd5671557e52cfb8721a1d23daf48f7e465b7c926561 WatchSource:0}: Error finding container fcb391e7dd2319314e28dd5671557e52cfb8721a1d23daf48f7e465b7c926561: Status 404 returned error can't find the container with id fcb391e7dd2319314e28dd5671557e52cfb8721a1d23daf48f7e465b7c926561 Oct 31 04:05:08.009189 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:08.009146 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" event={"ID":"925cd563-0f7b-439d-a607-7acd7326c922","Type":"ContainerStarted","Data":"f4c59251e8397167fe6a075a3d41254a231cfb834242ba3963dff6ae91f9b4e8"} Oct 31 04:05:08.009668 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:08.009218 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" event={"ID":"925cd563-0f7b-439d-a607-7acd7326c922","Type":"ContainerStarted","Data":"fcb391e7dd2319314e28dd5671557e52cfb8721a1d23daf48f7e465b7c926561"} Oct 31 04:05:08.123558 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:08.123482 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" podStartSLOduration=2.12346289 podStartE2EDuration="2.12346289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:08.119178163 +0000 UTC m=+2205.119803390" watchObservedRunningTime="2026-10-31 04:05:08.12346289 +0000 UTC m=+2205.124088118" Oct 31 04:05:09.019815 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:09.019773 4426 generic.go:356] "Generic (PLEG): container finished" podID="925cd563-0f7b-439d-a607-7acd7326c922" containerID="f4c59251e8397167fe6a075a3d41254a231cfb834242ba3963dff6ae91f9b4e8" exitCode=0 Oct 31 04:05:09.020315 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:09.019867 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" event={"ID":"925cd563-0f7b-439d-a607-7acd7326c922","Type":"ContainerDied","Data":"f4c59251e8397167fe6a075a3d41254a231cfb834242ba3963dff6ae91f9b4e8"} Oct 31 04:05:10.200815 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:10.200787 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:10.239406 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:10.239366 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-xks-ca\") pod \"925cd563-0f7b-439d-a607-7acd7326c922\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " Oct 31 04:05:10.239593 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:10.239456 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925cd563-0f7b-439d-a607-7acd7326c922-kube-api-access-9t2wh\" (UniqueName: \"kubernetes.io/projected/925cd563-0f7b-439d-a607-7acd7326c922-kube-api-access-9t2wh\") pod \"925cd563-0f7b-439d-a607-7acd7326c922\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " Oct 31 04:05:10.239637 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:10.239594 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-openshift-ca\") pod \"925cd563-0f7b-439d-a607-7acd7326c922\" (UID: \"925cd563-0f7b-439d-a607-7acd7326c922\") " Oct 31 04:05:10.240013 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:10.239977 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-openshift-ca" pod "925cd563-0f7b-439d-a607-7acd7326c922" (UID: "925cd563-0f7b-439d-a607-7acd7326c922"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:10.240392 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:10.240355 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:10.241687 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:10.241662 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-xks-ca" pod "925cd563-0f7b-439d-a607-7acd7326c922" (UID: "925cd563-0f7b-439d-a607-7acd7326c922"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:05:10.241798 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:10.241706 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925cd563-0f7b-439d-a607-7acd7326c922-kube-api-access-9t2wh" pod "925cd563-0f7b-439d-a607-7acd7326c922" (UID: "925cd563-0f7b-439d-a607-7acd7326c922"). InnerVolumeSpecName "kube-api-access-9t2wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:05:10.342416 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:10.342323 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925cd563-0f7b-439d-a607-7acd7326c922-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:10.342416 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:10.342355 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2wh\" (UniqueName: \"kubernetes.io/projected/925cd563-0f7b-439d-a607-7acd7326c922-kube-api-access-9t2wh\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:11.044598 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:11.044568 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" Oct 31 04:05:11.044787 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:11.044595 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb" event={"ID":"925cd563-0f7b-439d-a607-7acd7326c922","Type":"ContainerDied","Data":"fcb391e7dd2319314e28dd5671557e52cfb8721a1d23daf48f7e465b7c926561"} Oct 31 04:05:11.044787 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:11.044662 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcb391e7dd2319314e28dd5671557e52cfb8721a1d23daf48f7e465b7c926561" Oct 31 04:05:18.428928 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.428887 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4"] Oct 31 04:05:18.429913 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.429894 4426 cpu_manager.go:414] "containerMap: removing container" podUID="925cd563-0f7b-439d-a607-7acd7326c922" containerName="revoke-keys" Oct 31 04:05:18.429964 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.429915 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925cd563-0f7b-439d-a607-7acd7326c922" containerName="revoke-keys" Oct 31 04:05:18.430137 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.430111 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="925cd563-0f7b-439d-a607-7acd7326c922" containerName="revoke-keys" Oct 31 04:05:18.435546 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.435526 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:18.502729 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.502694 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4"] Oct 31 04:05:18.537417 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.537381 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:05:18.537770 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.537747 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:05:18.538056 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.538036 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:05:18.538874 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.538852 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:05:18.567957 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.567908 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4cp\" (UniqueName: \"kubernetes.io/projected/92b304e4-6937-490d-82c1-24dafd31b2bc-kube-api-access-bd4cp\") pod \"maas-api-revoke-sub-8cf615e8-nmkf4\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:18.568171 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.568010 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cf615e8-nmkf4\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:18.568171 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.568095 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cf615e8-nmkf4\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:18.669075 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.669038 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cf615e8-nmkf4\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:18.669297 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.669247 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4cp\" (UniqueName: \"kubernetes.io/projected/92b304e4-6937-490d-82c1-24dafd31b2bc-kube-api-access-bd4cp\") pod \"maas-api-revoke-sub-8cf615e8-nmkf4\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:18.669384 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.669350 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cf615e8-nmkf4\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:18.670050 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.670024 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cf615e8-nmkf4\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:18.671524 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.671499 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cf615e8-nmkf4\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:18.727388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.727296 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4cp\" (UniqueName: \"kubernetes.io/projected/92b304e4-6937-490d-82c1-24dafd31b2bc-kube-api-access-bd4cp\") pod \"maas-api-revoke-sub-8cf615e8-nmkf4\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:18.750443 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:18.750401 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:19.136238 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:19.136150 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4"] Oct 31 04:05:19.140255 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:05:19.140226 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b304e4_6937_490d_82c1_24dafd31b2bc.slice/crio-6a88b998df7ac4316ac8d4a1746f2bf77ab143085495267a80f5470987f2e711 WatchSource:0}: Error finding container 6a88b998df7ac4316ac8d4a1746f2bf77ab143085495267a80f5470987f2e711: Status 404 returned error can't find the container with id 6a88b998df7ac4316ac8d4a1746f2bf77ab143085495267a80f5470987f2e711 Oct 31 04:05:20.130011 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:20.129973 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" event={"ID":"92b304e4-6937-490d-82c1-24dafd31b2bc","Type":"ContainerStarted","Data":"3db59308214ae53fe95cba62b04c643fb67888a3a1603a1439e092a7a1c8308d"} Oct 31 04:05:20.130467 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:20.130039 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" event={"ID":"92b304e4-6937-490d-82c1-24dafd31b2bc","Type":"ContainerStarted","Data":"6a88b998df7ac4316ac8d4a1746f2bf77ab143085495267a80f5470987f2e711"} Oct 31 04:05:20.241191 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:20.241108 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" podStartSLOduration=2.241093196 podStartE2EDuration="2.241093196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:20.238758784 +0000 UTC m=+2217.239384013" watchObservedRunningTime="2026-10-31 04:05:20.241093196 +0000 UTC m=+2217.241718423" Oct 31 04:05:21.141231 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:21.141192 4426 generic.go:356] "Generic (PLEG): container finished" podID="92b304e4-6937-490d-82c1-24dafd31b2bc" containerID="3db59308214ae53fe95cba62b04c643fb67888a3a1603a1439e092a7a1c8308d" exitCode=0 Oct 31 04:05:21.141686 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:21.141260 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" event={"ID":"92b304e4-6937-490d-82c1-24dafd31b2bc","Type":"ContainerDied","Data":"3db59308214ae53fe95cba62b04c643fb67888a3a1603a1439e092a7a1c8308d"} Oct 31 04:05:22.342373 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:22.342296 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:22.524400 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:22.524311 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-openshift-ca\") pod \"92b304e4-6937-490d-82c1-24dafd31b2bc\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " Oct 31 04:05:22.524574 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:22.524496 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-xks-ca\") pod \"92b304e4-6937-490d-82c1-24dafd31b2bc\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " Oct 31 04:05:22.524574 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:22.524530 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b304e4-6937-490d-82c1-24dafd31b2bc-kube-api-access-bd4cp\" (UniqueName: \"kubernetes.io/projected/92b304e4-6937-490d-82c1-24dafd31b2bc-kube-api-access-bd4cp\") pod \"92b304e4-6937-490d-82c1-24dafd31b2bc\" (UID: \"92b304e4-6937-490d-82c1-24dafd31b2bc\") " Oct 31 04:05:22.524735 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:22.524702 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-openshift-ca" pod "92b304e4-6937-490d-82c1-24dafd31b2bc" (UID: "92b304e4-6937-490d-82c1-24dafd31b2bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:22.525099 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:22.525075 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:22.527191 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:22.527159 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b304e4-6937-490d-82c1-24dafd31b2bc-kube-api-access-bd4cp" pod "92b304e4-6937-490d-82c1-24dafd31b2bc" (UID: "92b304e4-6937-490d-82c1-24dafd31b2bc"). InnerVolumeSpecName "kube-api-access-bd4cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:05:22.527326 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:22.527272 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-xks-ca" pod "92b304e4-6937-490d-82c1-24dafd31b2bc" (UID: "92b304e4-6937-490d-82c1-24dafd31b2bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:05:22.626802 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:22.626763 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b304e4-6937-490d-82c1-24dafd31b2bc-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:22.626802 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:22.626797 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd4cp\" (UniqueName: \"kubernetes.io/projected/92b304e4-6937-490d-82c1-24dafd31b2bc-kube-api-access-bd4cp\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:23.159320 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:23.159281 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" event={"ID":"92b304e4-6937-490d-82c1-24dafd31b2bc","Type":"ContainerDied","Data":"6a88b998df7ac4316ac8d4a1746f2bf77ab143085495267a80f5470987f2e711"} Oct 31 04:05:23.159320 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:23.159324 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4" Oct 31 04:05:23.159575 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:23.159376 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a88b998df7ac4316ac8d4a1746f2bf77ab143085495267a80f5470987f2e711" Oct 31 04:05:25.755141 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:25.755097 4426 scope.go:122] "RemoveContainer" containerID="b7614ff91c8144bdfd16078c9107b7a8077c2e565ee15104c5408848bded5193" Oct 31 04:05:32.851843 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:05:32.851796 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:05:35.224837 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:35.224799 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:05:43.719883 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:43.719832 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp"] Oct 31 04:05:43.720438 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:43.720111 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" containerID="cri-o://ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816" gracePeriod=30 Oct 31 04:05:45.310025 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:45.309987 4426 patch_prober.go:29] interesting pod/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:05:45.310511 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:45.310055 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:05:49.049539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.049508 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 04:05:49.111899 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.111864 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89149831-510f-4dbb-8633-63176576a247-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89149831-510f-4dbb-8633-63176576a247-istiod-ca-cert\") pod \"89149831-510f-4dbb-8633-63176576a247\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " Oct 31 04:05:49.112085 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.111919 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-socket\") pod \"89149831-510f-4dbb-8633-63176576a247\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " Oct 31 04:05:49.112085 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.111953 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-kube-api-access-s8qx2\" (UniqueName: \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-kube-api-access-s8qx2\") pod \"89149831-510f-4dbb-8633-63176576a247\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " Oct 31 04:05:49.112085 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.112059 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-data\") pod \"89149831-510f-4dbb-8633-63176576a247\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " Oct 31 04:05:49.112325 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.112206 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89149831-510f-4dbb-8633-63176576a247-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89149831-510f-4dbb-8633-63176576a247-istio-podinfo\") pod \"89149831-510f-4dbb-8633-63176576a247\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " Oct 31 04:05:49.112325 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.112301 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-certs\") pod \"89149831-510f-4dbb-8633-63176576a247\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " Oct 31 04:05:49.112431 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.112348 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-socket" pod "89149831-510f-4dbb-8633-63176576a247" (UID: "89149831-510f-4dbb-8633-63176576a247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:05:49.112431 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.112361 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-envoy\") pod \"89149831-510f-4dbb-8633-63176576a247\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " Oct 31 04:05:49.112431 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.112379 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-data" pod "89149831-510f-4dbb-8633-63176576a247" (UID: "89149831-510f-4dbb-8633-63176576a247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:05:49.113099 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.112450 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89149831-510f-4dbb-8633-63176576a247-istiod-ca-cert" pod "89149831-510f-4dbb-8633-63176576a247" (UID: "89149831-510f-4dbb-8633-63176576a247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:49.113099 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.112465 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-credential-socket\") pod \"89149831-510f-4dbb-8633-63176576a247\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " Oct 31 04:05:49.113099 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.112506 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-istio-token\" (UniqueName: \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-istio-token\") pod \"89149831-510f-4dbb-8633-63176576a247\" (UID: \"89149831-510f-4dbb-8633-63176576a247\") " Oct 31 04:05:49.113099 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.112598 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-certs" pod "89149831-510f-4dbb-8633-63176576a247" (UID: "89149831-510f-4dbb-8633-63176576a247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:05:49.113099 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.112745 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-credential-socket" pod "89149831-510f-4dbb-8633-63176576a247" (UID: "89149831-510f-4dbb-8633-63176576a247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:05:49.113452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.113300 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:49.113452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.113328 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:49.113452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.113345 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:49.113452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.113359 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89149831-510f-4dbb-8633-63176576a247-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:49.113452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.113374 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:49.114775 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.114513 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-envoy" pod "89149831-510f-4dbb-8633-63176576a247" (UID: "89149831-510f-4dbb-8633-63176576a247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:05:49.115306 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.115280 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89149831-510f-4dbb-8633-63176576a247-istio-podinfo" pod "89149831-510f-4dbb-8633-63176576a247" (UID: "89149831-510f-4dbb-8633-63176576a247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:05:49.115434 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.115317 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-kube-api-access-s8qx2" pod "89149831-510f-4dbb-8633-63176576a247" (UID: "89149831-510f-4dbb-8633-63176576a247"). InnerVolumeSpecName "kube-api-access-s8qx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:05:49.115434 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.115397 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-istio-token" pod "89149831-510f-4dbb-8633-63176576a247" (UID: "89149831-510f-4dbb-8633-63176576a247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:05:49.219199 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.215228 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8qx2\" (UniqueName: \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-kube-api-access-s8qx2\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:49.219199 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.215262 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89149831-510f-4dbb-8633-63176576a247-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:49.219199 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.215278 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89149831-510f-4dbb-8633-63176576a247-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:49.219199 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.215303 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89149831-510f-4dbb-8633-63176576a247-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:05:49.398355 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.398319 4426 generic.go:356] "Generic (PLEG): container finished" podID="89149831-510f-4dbb-8633-63176576a247" containerID="ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816" exitCode=0 Oct 31 04:05:49.398571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.398384 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" Oct 31 04:05:49.398571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.398396 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" event={"ID":"89149831-510f-4dbb-8633-63176576a247","Type":"ContainerDied","Data":"ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816"} Oct 31 04:05:49.398571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.398449 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp" event={"ID":"89149831-510f-4dbb-8633-63176576a247","Type":"ContainerDied","Data":"7528ae6b42232d3b22e68bf1cc57828a1d8893e92beb93d2a4ea139ecd482071"} Oct 31 04:05:49.398571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.398472 4426 scope.go:122] "RemoveContainer" containerID="ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816" Oct 31 04:05:49.412734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.412627 4426 scope.go:122] "RemoveContainer" containerID="ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816" Oct 31 04:05:49.413611 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:05:49.413515 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816\": container with ID starting with ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816 not found: ID does not exist" containerID="ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816" Oct 31 04:05:49.413611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.413557 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816"} err="failed to get container status \"ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816\": rpc error: code = NotFound desc = could not find container \"ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816\": container with ID starting with ee9744d723db3489433151f5394c917458b16b6976a44e1bfa5966d7ca9de816 not found: ID does not exist" Oct 31 04:05:49.564864 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.564824 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp"] Oct 31 04:05:49.569535 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:49.569505 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-lk8sp"] Oct 31 04:05:51.137222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:51.137179 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89149831-510f-4dbb-8633-63176576a247" path="/var/lib/kubelet/pods/89149831-510f-4dbb-8633-63176576a247/volumes" Oct 31 04:05:58.758151 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:58.757645 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t"] Oct 31 04:05:58.758151 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:58.757909 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" containerID="cri-o://8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f" gracePeriod=30 Oct 31 04:05:58.808228 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:58.808188 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h"] Oct 31 04:05:58.808484 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:05:58.808456 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" containerID="cri-o://6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787" gracePeriod=30 Oct 31 04:06:01.378440 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:01.378400 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6"] Oct 31 04:06:01.427354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:01.427314 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68aa3e-htkt6"] Oct 31 04:06:02.040713 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:02.040679 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:06:02.040886 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:02.040747 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:06:02.356745 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:02.356663 4426 patch_prober.go:29] interesting pod/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:06:02.356745 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:02.356729 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:06:02.847249 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:06:02.847209 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:06:03.132464 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:03.132384 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eaf0207-9a3d-449c-829f-d7437958dc5b" path="/var/lib/kubelet/pods/7eaf0207-9a3d-449c-829f-d7437958dc5b/volumes" Oct 31 04:06:04.112919 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.112891 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 04:06:04.149657 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.149630 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 04:06:04.259215 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259180 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-envoy\") pod \"7c2f7796-6471-497b-a49a-5c13e85e4213\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " Oct 31 04:06:04.259424 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259232 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-istio-token\" (UniqueName: \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-istio-token\") pod \"44dd1297-446c-439e-99aa-87097742d386\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " Oct 31 04:06:04.259424 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259270 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-socket\") pod \"7c2f7796-6471-497b-a49a-5c13e85e4213\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " Oct 31 04:06:04.259424 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259333 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-certs\") pod \"7c2f7796-6471-497b-a49a-5c13e85e4213\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " Oct 31 04:06:04.259424 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259394 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2f7796-6471-497b-a49a-5c13e85e4213-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2f7796-6471-497b-a49a-5c13e85e4213-istiod-ca-cert\") pod \"7c2f7796-6471-497b-a49a-5c13e85e4213\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " Oct 31 04:06:04.259424 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259423 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-data\") pod \"44dd1297-446c-439e-99aa-87097742d386\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " Oct 31 04:06:04.259713 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259460 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44dd1297-446c-439e-99aa-87097742d386-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44dd1297-446c-439e-99aa-87097742d386-istiod-ca-cert\") pod \"44dd1297-446c-439e-99aa-87097742d386\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " Oct 31 04:06:04.259713 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259529 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-credential-socket\") pod \"7c2f7796-6471-497b-a49a-5c13e85e4213\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " Oct 31 04:06:04.259713 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259562 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-data\") pod \"7c2f7796-6471-497b-a49a-5c13e85e4213\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " Oct 31 04:06:04.259713 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259632 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-podinfo\") pod \"7c2f7796-6471-497b-a49a-5c13e85e4213\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " Oct 31 04:06:04.259713 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259647 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-certs" pod "7c2f7796-6471-497b-a49a-5c13e85e4213" (UID: "7c2f7796-6471-497b-a49a-5c13e85e4213"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:04.259713 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259675 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-socket\") pod \"44dd1297-446c-439e-99aa-87097742d386\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " Oct 31 04:06:04.259713 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259707 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-credential-socket\") pod \"44dd1297-446c-439e-99aa-87097742d386\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " Oct 31 04:06:04.260096 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259794 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-kube-api-access-hgrrc\" (UniqueName: \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-kube-api-access-hgrrc\") pod \"7c2f7796-6471-497b-a49a-5c13e85e4213\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " Oct 31 04:06:04.260096 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.259946 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-envoy\") pod \"44dd1297-446c-439e-99aa-87097742d386\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " Oct 31 04:06:04.260096 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.260002 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-kube-api-access-9svb8\" (UniqueName: \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-kube-api-access-9svb8\") pod \"44dd1297-446c-439e-99aa-87097742d386\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " Oct 31 04:06:04.260096 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.260036 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44dd1297-446c-439e-99aa-87097742d386-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44dd1297-446c-439e-99aa-87097742d386-istio-podinfo\") pod \"44dd1297-446c-439e-99aa-87097742d386\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " Oct 31 04:06:04.260096 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.260056 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-socket" pod "44dd1297-446c-439e-99aa-87097742d386" (UID: "44dd1297-446c-439e-99aa-87097742d386"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:04.260096 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.260063 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-token\" (UniqueName: \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-token\") pod \"7c2f7796-6471-497b-a49a-5c13e85e4213\" (UID: \"7c2f7796-6471-497b-a49a-5c13e85e4213\") " Oct 31 04:06:04.260456 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.260169 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-credential-socket" pod "44dd1297-446c-439e-99aa-87097742d386" (UID: "44dd1297-446c-439e-99aa-87097742d386"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:04.260456 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.260175 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-certs\") pod \"44dd1297-446c-439e-99aa-87097742d386\" (UID: \"44dd1297-446c-439e-99aa-87097742d386\") " Oct 31 04:06:04.260456 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.260360 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-data" pod "44dd1297-446c-439e-99aa-87097742d386" (UID: "44dd1297-446c-439e-99aa-87097742d386"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:04.260456 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.260390 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-data" pod "7c2f7796-6471-497b-a49a-5c13e85e4213" (UID: "7c2f7796-6471-497b-a49a-5c13e85e4213"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:04.260456 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.260394 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-certs" pod "44dd1297-446c-439e-99aa-87097742d386" (UID: "44dd1297-446c-439e-99aa-87097742d386"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:04.260456 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.260410 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-socket" pod "7c2f7796-6471-497b-a49a-5c13e85e4213" (UID: "7c2f7796-6471-497b-a49a-5c13e85e4213"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:04.260804 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.260745 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44dd1297-446c-439e-99aa-87097742d386-istiod-ca-cert" pod "44dd1297-446c-439e-99aa-87097742d386" (UID: "44dd1297-446c-439e-99aa-87097742d386"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:04.261537 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.261498 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-credential-socket" pod "7c2f7796-6471-497b-a49a-5c13e85e4213" (UID: "7c2f7796-6471-497b-a49a-5c13e85e4213"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:04.261537 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.261526 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.261761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.261552 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.261761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.261568 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.261761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.261582 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.261761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.261601 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44dd1297-446c-439e-99aa-87097742d386-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.261761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.261616 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.261761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.261632 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.261761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.261635 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2f7796-6471-497b-a49a-5c13e85e4213-istiod-ca-cert" pod "7c2f7796-6471-497b-a49a-5c13e85e4213" (UID: "7c2f7796-6471-497b-a49a-5c13e85e4213"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:04.261761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.261647 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.262331 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.262033 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-envoy" pod "7c2f7796-6471-497b-a49a-5c13e85e4213" (UID: "7c2f7796-6471-497b-a49a-5c13e85e4213"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:04.263602 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.263579 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-istio-token" pod "44dd1297-446c-439e-99aa-87097742d386" (UID: "44dd1297-446c-439e-99aa-87097742d386"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:04.263716 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.263674 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-podinfo" pod "7c2f7796-6471-497b-a49a-5c13e85e4213" (UID: "7c2f7796-6471-497b-a49a-5c13e85e4213"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:06:04.263716 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.263704 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-envoy" pod "44dd1297-446c-439e-99aa-87097742d386" (UID: "44dd1297-446c-439e-99aa-87097742d386"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:04.263840 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.263773 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-token" pod "7c2f7796-6471-497b-a49a-5c13e85e4213" (UID: "7c2f7796-6471-497b-a49a-5c13e85e4213"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:04.264009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.263980 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-kube-api-access-9svb8" pod "44dd1297-446c-439e-99aa-87097742d386" (UID: "44dd1297-446c-439e-99aa-87097742d386"). InnerVolumeSpecName "kube-api-access-9svb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:04.264119 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.264053 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-kube-api-access-hgrrc" pod "7c2f7796-6471-497b-a49a-5c13e85e4213" (UID: "7c2f7796-6471-497b-a49a-5c13e85e4213"). InnerVolumeSpecName "kube-api-access-hgrrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:04.264467 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.264439 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44dd1297-446c-439e-99aa-87097742d386-istio-podinfo" pod "44dd1297-446c-439e-99aa-87097742d386" (UID: "44dd1297-446c-439e-99aa-87097742d386"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:06:04.362869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.362775 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.362869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.362812 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.362869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.362822 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgrrc\" (UniqueName: \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-kube-api-access-hgrrc\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.362869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.362832 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44dd1297-446c-439e-99aa-87097742d386-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.362869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.362842 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9svb8\" (UniqueName: \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-kube-api-access-9svb8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.362869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.362851 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44dd1297-446c-439e-99aa-87097742d386-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.362869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.362861 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.362869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.362870 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2f7796-6471-497b-a49a-5c13e85e4213-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.362869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.362879 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44dd1297-446c-439e-99aa-87097742d386-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.363365 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.362889 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2f7796-6471-497b-a49a-5c13e85e4213-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.577452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.577410 4426 generic.go:356] "Generic (PLEG): container finished" podID="44dd1297-446c-439e-99aa-87097742d386" containerID="6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787" exitCode=0 Oct 31 04:06:04.577656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.577476 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" Oct 31 04:06:04.577656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.577528 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" event={"ID":"44dd1297-446c-439e-99aa-87097742d386","Type":"ContainerDied","Data":"6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787"} Oct 31 04:06:04.577656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.577580 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h" event={"ID":"44dd1297-446c-439e-99aa-87097742d386","Type":"ContainerDied","Data":"fc80523a0e243340fcf31a52b224b1dda00c34126ab10910e6408040be6f86c9"} Oct 31 04:06:04.577656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.577610 4426 scope.go:122] "RemoveContainer" containerID="6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787" Oct 31 04:06:04.579562 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.579527 4426 generic.go:356] "Generic (PLEG): container finished" podID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerID="8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f" exitCode=0 Oct 31 04:06:04.579695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.579643 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" event={"ID":"7c2f7796-6471-497b-a49a-5c13e85e4213","Type":"ContainerDied","Data":"8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f"} Oct 31 04:06:04.579695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.579687 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" event={"ID":"7c2f7796-6471-497b-a49a-5c13e85e4213","Type":"ContainerDied","Data":"02040eeb92e14de37985f9ac642e6b0bbd7a4acd21ec3fd97678970d7b51dd5b"} Oct 31 04:06:04.580326 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.580302 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t" Oct 31 04:06:04.590884 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.590861 4426 scope.go:122] "RemoveContainer" containerID="6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787" Oct 31 04:06:04.591590 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:06:04.591234 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787\": container with ID starting with 6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787 not found: ID does not exist" containerID="6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787" Oct 31 04:06:04.591590 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.591270 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787"} err="failed to get container status \"6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787\": rpc error: code = NotFound desc = could not find container \"6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787\": container with ID starting with 6e5e103dd8137dcb58062f02121d60198a5de51c063a4986fc5b7586bbfce787 not found: ID does not exist" Oct 31 04:06:04.591590 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.591294 4426 scope.go:122] "RemoveContainer" containerID="8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f" Oct 31 04:06:04.605690 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.605668 4426 scope.go:122] "RemoveContainer" containerID="8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f" Oct 31 04:06:04.606023 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:06:04.605969 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f\": container with ID starting with 8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f not found: ID does not exist" containerID="8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f" Oct 31 04:06:04.606023 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.606003 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f"} err="failed to get container status \"8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f\": rpc error: code = NotFound desc = could not find container \"8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f\": container with ID starting with 8e192a1800f00c7371dbbb0c1cb15e9a7217ac2a800abdc12d7409c560bbf10f not found: ID does not exist" Oct 31 04:06:04.738910 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.738878 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h"] Oct 31 04:06:04.744165 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.744119 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-qnm8h"] Oct 31 04:06:04.846180 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.846095 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t"] Oct 31 04:06:04.851307 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:04.851272 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-pzg4t"] Oct 31 04:06:05.131215 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:05.131097 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44dd1297-446c-439e-99aa-87097742d386" path="/var/lib/kubelet/pods/44dd1297-446c-439e-99aa-87097742d386/volumes" Oct 31 04:06:05.132377 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:05.132345 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" path="/var/lib/kubelet/pods/7c2f7796-6471-497b-a49a-5c13e85e4213/volumes" Oct 31 04:06:08.160320 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:08.160287 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m"] Oct 31 04:06:08.160856 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:08.160527 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" containerID="cri-o://d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7" gracePeriod=30 Oct 31 04:06:08.165409 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:08.165382 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6"] Oct 31 04:06:08.165637 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:08.165600 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerName="istio-proxy" containerID="cri-o://808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7" gracePeriod=30 Oct 31 04:06:09.623550 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:09.623509 4426 patch_prober.go:29] interesting pod/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:06:09.623994 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:09.623591 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:06:11.954170 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:11.954115 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5"] Oct 31 04:06:12.000264 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:12.000229 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f9a7fa-dh8n5"] Oct 31 04:06:13.132865 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.132827 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595ba575-a34a-4a82-a76d-4c6b20e06bfd" path="/var/lib/kubelet/pods/595ba575-a34a-4a82-a76d-4c6b20e06bfd/volumes" Oct 31 04:06:13.467558 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.467520 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 04:06:13.512638 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.512612 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 04:06:13.514835 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.514808 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-kube-api-access-zjq6z\" (UniqueName: \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-kube-api-access-zjq6z\") pod \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " Oct 31 04:06:13.514964 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.514870 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-token\" (UniqueName: \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-token\") pod \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " Oct 31 04:06:13.514964 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.514956 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-envoy\") pod \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " Oct 31 04:06:13.515061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.514980 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istiod-ca-cert\") pod \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " Oct 31 04:06:13.515061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.515013 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-socket\") pod \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " Oct 31 04:06:13.515061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.515034 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-data\") pod \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " Oct 31 04:06:13.515261 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.515089 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-certs\") pod \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " Oct 31 04:06:13.515261 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.515156 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-podinfo\") pod \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " Oct 31 04:06:13.515384 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.515261 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-credential-socket\") pod \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\" (UID: \"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e\") " Oct 31 04:06:13.515384 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.515316 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-socket" pod "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" (UID: "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:13.515570 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.515545 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istiod-ca-cert" pod "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" (UID: "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:13.515645 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.515566 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-certs" pod "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" (UID: "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:13.515766 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.515743 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-data" pod "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" (UID: "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:13.515766 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.515755 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-credential-socket" pod "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" (UID: "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:13.516031 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.516002 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.516185 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.516034 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.516185 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.516050 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.516185 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.516065 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.516185 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.516079 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.517520 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.517498 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-token" pod "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" (UID: "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:13.517760 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.517733 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-podinfo" pod "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" (UID: "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:06:13.518214 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.518190 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-envoy" pod "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" (UID: "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:13.518289 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.518274 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-kube-api-access-zjq6z" pod "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" (UID: "8ba30cf5-bad6-4938-a1d4-c125dc8ac20e"). InnerVolumeSpecName "kube-api-access-zjq6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:13.617306 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.617273 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-token\" (UniqueName: \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-token\") pod \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " Oct 31 04:06:13.617518 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.617355 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-certs\") pod \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " Oct 31 04:06:13.617518 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.617424 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-kube-api-access-jnhvq\" (UniqueName: \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-kube-api-access-jnhvq\") pod \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " Oct 31 04:06:13.617518 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.617507 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-data\") pod \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.617579 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-socket\") pod \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.617634 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istiod-ca-cert\") pod \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.617664 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-credential-socket\") pod \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.617675 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-certs" pod "3bb4b041-8a0b-4e29-a276-a9ee7039a581" (UID: "3bb4b041-8a0b-4e29-a276-a9ee7039a581"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.617753 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-podinfo\") pod \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.617789 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-envoy\") pod \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\" (UID: \"3bb4b041-8a0b-4e29-a276-a9ee7039a581\") " Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.617987 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-credential-socket" pod "3bb4b041-8a0b-4e29-a276-a9ee7039a581" (UID: "3bb4b041-8a0b-4e29-a276-a9ee7039a581"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.618408 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istiod-ca-cert" pod "3bb4b041-8a0b-4e29-a276-a9ee7039a581" (UID: "3bb4b041-8a0b-4e29-a276-a9ee7039a581"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.618475 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-socket" pod "3bb4b041-8a0b-4e29-a276-a9ee7039a581" (UID: "3bb4b041-8a0b-4e29-a276-a9ee7039a581"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.618512 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.618554 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.618571 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.618588 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjq6z\" (UniqueName: \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-kube-api-access-zjq6z\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.618605 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.618621 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.618908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.618675 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-data" pod "3bb4b041-8a0b-4e29-a276-a9ee7039a581" (UID: "3bb4b041-8a0b-4e29-a276-a9ee7039a581"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:13.620075 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.620050 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-token" pod "3bb4b041-8a0b-4e29-a276-a9ee7039a581" (UID: "3bb4b041-8a0b-4e29-a276-a9ee7039a581"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:13.620568 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.620547 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-kube-api-access-jnhvq" pod "3bb4b041-8a0b-4e29-a276-a9ee7039a581" (UID: "3bb4b041-8a0b-4e29-a276-a9ee7039a581"). InnerVolumeSpecName "kube-api-access-jnhvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:13.620636 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.620588 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-envoy" pod "3bb4b041-8a0b-4e29-a276-a9ee7039a581" (UID: "3bb4b041-8a0b-4e29-a276-a9ee7039a581"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:13.621440 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.621418 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-podinfo" pod "3bb4b041-8a0b-4e29-a276-a9ee7039a581" (UID: "3bb4b041-8a0b-4e29-a276-a9ee7039a581"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:06:13.677084 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.677052 4426 generic.go:356] "Generic (PLEG): container finished" podID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerID="d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7" exitCode=0 Oct 31 04:06:13.677335 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.677102 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" Oct 31 04:06:13.677335 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.677137 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" event={"ID":"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e","Type":"ContainerDied","Data":"d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7"} Oct 31 04:06:13.677335 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.677198 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m" event={"ID":"8ba30cf5-bad6-4938-a1d4-c125dc8ac20e","Type":"ContainerDied","Data":"703a34d938cdf59ce9edba4f68f84c54ff7c7a147fb1262c636753e4ef5d136f"} Oct 31 04:06:13.677335 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.677228 4426 scope.go:122] "RemoveContainer" containerID="d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7" Oct 31 04:06:13.679069 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.679039 4426 generic.go:356] "Generic (PLEG): container finished" podID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerID="808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7" exitCode=0 Oct 31 04:06:13.679442 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.679418 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" event={"ID":"3bb4b041-8a0b-4e29-a276-a9ee7039a581","Type":"ContainerDied","Data":"808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7"} Oct 31 04:06:13.679497 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.679480 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" event={"ID":"3bb4b041-8a0b-4e29-a276-a9ee7039a581","Type":"ContainerDied","Data":"b7eb03b16e3e58fb384fa1bf390e1c487f0844f0750aad15f39c349c847e1f16"} Oct 31 04:06:13.679582 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.679567 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6" Oct 31 04:06:13.690101 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.690082 4426 scope.go:122] "RemoveContainer" containerID="d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7" Oct 31 04:06:13.690409 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:06:13.690386 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7\": container with ID starting with d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7 not found: ID does not exist" containerID="d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7" Oct 31 04:06:13.690466 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.690419 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7"} err="failed to get container status \"d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7\": rpc error: code = NotFound desc = could not find container \"d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7\": container with ID starting with d41dbabdd8a276bbe3d16ec154298a06a89824321462bd55d9b8d6fe81c663f7 not found: ID does not exist" Oct 31 04:06:13.690466 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.690438 4426 scope.go:122] "RemoveContainer" containerID="808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7" Oct 31 04:06:13.700903 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.700883 4426 scope.go:122] "RemoveContainer" containerID="808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7" Oct 31 04:06:13.701212 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:06:13.701186 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7\": container with ID starting with 808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7 not found: ID does not exist" containerID="808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7" Oct 31 04:06:13.701310 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.701217 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7"} err="failed to get container status \"808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7\": rpc error: code = NotFound desc = could not find container \"808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7\": container with ID starting with 808bac9881d95036319e63ef8cd2c816867e0ff7c3fbd56a4e8c6d548bbfa1a7 not found: ID does not exist" Oct 31 04:06:13.719760 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.719729 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnhvq\" (UniqueName: \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-kube-api-access-jnhvq\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.719760 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.719763 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.719975 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.719781 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.719975 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.719795 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.719975 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.719806 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.719975 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.719820 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.719975 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.719834 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb4b041-8a0b-4e29-a276-a9ee7039a581-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:13.844616 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.844573 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6"] Oct 31 04:06:13.849025 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.848991 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-8zpp6"] Oct 31 04:06:13.954204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.954083 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m"] Oct 31 04:06:13.960867 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:13.960830 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-68aa3e-openshift-default-5f5c8bdd5-n6r7m"] Oct 31 04:06:15.132357 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:15.132319 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" path="/var/lib/kubelet/pods/3bb4b041-8a0b-4e29-a276-a9ee7039a581/volumes" Oct 31 04:06:15.132895 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:15.132878 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" path="/var/lib/kubelet/pods/8ba30cf5-bad6-4938-a1d4-c125dc8ac20e/volumes" Oct 31 04:06:17.365396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:17.365357 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:06:20.316800 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.316762 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4"] Oct 31 04:06:20.317676 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317645 4426 cpu_manager.go:414] "containerMap: removing container" podUID="92b304e4-6937-490d-82c1-24dafd31b2bc" containerName="revoke-keys" Oct 31 04:06:20.317676 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317673 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b304e4-6937-490d-82c1-24dafd31b2bc" containerName="revoke-keys" Oct 31 04:06:20.317849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317720 4426 cpu_manager.go:414] "containerMap: removing container" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerName="istio-proxy" Oct 31 04:06:20.317849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317728 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerName="istio-proxy" Oct 31 04:06:20.317849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317749 4426 cpu_manager.go:414] "containerMap: removing container" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" Oct 31 04:06:20.317849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317756 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" Oct 31 04:06:20.317849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317773 4426 cpu_manager.go:414] "containerMap: removing container" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" Oct 31 04:06:20.317849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317783 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" Oct 31 04:06:20.317849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317798 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" Oct 31 04:06:20.317849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317806 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" Oct 31 04:06:20.317849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317823 4426 cpu_manager.go:414] "containerMap: removing container" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" Oct 31 04:06:20.317849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317830 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" Oct 31 04:06:20.318195 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317979 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb4b041-8a0b-4e29-a276-a9ee7039a581" containerName="istio-proxy" Oct 31 04:06:20.318195 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.317996 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba30cf5-bad6-4938-a1d4-c125dc8ac20e" containerName="istio-proxy" Oct 31 04:06:20.318195 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.318003 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="89149831-510f-4dbb-8633-63176576a247" containerName="istio-proxy" Oct 31 04:06:20.318195 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.318010 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2f7796-6471-497b-a49a-5c13e85e4213" containerName="istio-proxy" Oct 31 04:06:20.318195 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.318017 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="44dd1297-446c-439e-99aa-87097742d386" containerName="istio-proxy" Oct 31 04:06:20.318195 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.318024 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b304e4-6937-490d-82c1-24dafd31b2bc" containerName="revoke-keys" Oct 31 04:06:20.321553 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.321535 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:20.420154 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.419896 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:06:20.421584 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.420780 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:06:20.421584 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.420856 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:06:20.421584 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.420911 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:06:20.427166 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.424579 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4"] Oct 31 04:06:20.445230 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.445195 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwdkd\" (UniqueName: \"kubernetes.io/projected/16d61453-63b4-4275-97f2-8fd85055f80a-kube-api-access-wwdkd\") pod \"maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:20.445422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.445253 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:20.445422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.445363 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:20.547340 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.547301 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:20.547582 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.547565 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwdkd\" (UniqueName: \"kubernetes.io/projected/16d61453-63b4-4275-97f2-8fd85055f80a-kube-api-access-wwdkd\") pod \"maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:20.547698 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.547620 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:20.548544 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.548516 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:20.550115 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.550096 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:20.605610 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.605533 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwdkd\" (UniqueName: \"kubernetes.io/projected/16d61453-63b4-4275-97f2-8fd85055f80a-kube-api-access-wwdkd\") pod \"maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:20.636091 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:20.636057 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:21.028769 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:21.028734 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4"] Oct 31 04:06:21.032920 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:06:21.032888 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d61453_63b4_4275_97f2_8fd85055f80a.slice/crio-dcc40c53f3aeff381e83dd10d682148b709fb1449189101aeff3c34e8b45a5b7 WatchSource:0}: Error finding container dcc40c53f3aeff381e83dd10d682148b709fb1449189101aeff3c34e8b45a5b7: Status 404 returned error can't find the container with id dcc40c53f3aeff381e83dd10d682148b709fb1449189101aeff3c34e8b45a5b7 Oct 31 04:06:21.757664 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:21.757609 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" event={"ID":"16d61453-63b4-4275-97f2-8fd85055f80a","Type":"ContainerStarted","Data":"d787a9f62171f777c6cd9e41df538aebbb0813c2f6f04c5882eb27a6fc6ff375"} Oct 31 04:06:21.758212 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:21.757677 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" event={"ID":"16d61453-63b4-4275-97f2-8fd85055f80a","Type":"ContainerStarted","Data":"dcc40c53f3aeff381e83dd10d682148b709fb1449189101aeff3c34e8b45a5b7"} Oct 31 04:06:21.868493 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:21.868421 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" podStartSLOduration=1.8684011470000002 podStartE2EDuration="1.868401147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:21.866090487 +0000 UTC m=+2278.866715718" watchObservedRunningTime="2026-10-31 04:06:21.868401147 +0000 UTC m=+2278.869026377" Oct 31 04:06:22.750553 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:22.750514 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb"] Oct 31 04:06:22.768826 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:22.768790 4426 generic.go:356] "Generic (PLEG): container finished" podID="16d61453-63b4-4275-97f2-8fd85055f80a" containerID="d787a9f62171f777c6cd9e41df538aebbb0813c2f6f04c5882eb27a6fc6ff375" exitCode=0 Oct 31 04:06:22.769260 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:22.768879 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" event={"ID":"16d61453-63b4-4275-97f2-8fd85055f80a","Type":"ContainerDied","Data":"d787a9f62171f777c6cd9e41df538aebbb0813c2f6f04c5882eb27a6fc6ff375"} Oct 31 04:06:22.796353 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:22.796319 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-833977-zfbxb"] Oct 31 04:06:23.139463 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:23.139405 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925cd563-0f7b-439d-a607-7acd7326c922" path="/var/lib/kubelet/pods/925cd563-0f7b-439d-a607-7acd7326c922/volumes" Oct 31 04:06:23.958357 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:23.958319 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:24.101029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.100934 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-xks-ca\") pod \"16d61453-63b4-4275-97f2-8fd85055f80a\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " Oct 31 04:06:24.101228 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.101065 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d61453-63b4-4275-97f2-8fd85055f80a-kube-api-access-wwdkd\" (UniqueName: \"kubernetes.io/projected/16d61453-63b4-4275-97f2-8fd85055f80a-kube-api-access-wwdkd\") pod \"16d61453-63b4-4275-97f2-8fd85055f80a\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " Oct 31 04:06:24.101308 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.101278 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-openshift-ca\") pod \"16d61453-63b4-4275-97f2-8fd85055f80a\" (UID: \"16d61453-63b4-4275-97f2-8fd85055f80a\") " Oct 31 04:06:24.101652 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.101611 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-openshift-ca" pod "16d61453-63b4-4275-97f2-8fd85055f80a" (UID: "16d61453-63b4-4275-97f2-8fd85055f80a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:24.101990 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.101965 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:24.103428 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.103395 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d61453-63b4-4275-97f2-8fd85055f80a-kube-api-access-wwdkd" pod "16d61453-63b4-4275-97f2-8fd85055f80a" (UID: "16d61453-63b4-4275-97f2-8fd85055f80a"). InnerVolumeSpecName "kube-api-access-wwdkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:24.103687 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.103666 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-xks-ca" pod "16d61453-63b4-4275-97f2-8fd85055f80a" (UID: "16d61453-63b4-4275-97f2-8fd85055f80a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:06:24.203265 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.203225 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d61453-63b4-4275-97f2-8fd85055f80a-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:24.203265 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.203261 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwdkd\" (UniqueName: \"kubernetes.io/projected/16d61453-63b4-4275-97f2-8fd85055f80a-kube-api-access-wwdkd\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:24.788209 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.788170 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" Oct 31 04:06:24.788419 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.788341 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4" event={"ID":"16d61453-63b4-4275-97f2-8fd85055f80a","Type":"ContainerDied","Data":"dcc40c53f3aeff381e83dd10d682148b709fb1449189101aeff3c34e8b45a5b7"} Oct 31 04:06:24.788419 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.788397 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcc40c53f3aeff381e83dd10d682148b709fb1449189101aeff3c34e8b45a5b7" Oct 31 04:06:24.794053 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.794021 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4"] Oct 31 04:06:24.841254 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.841218 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj"] Oct 31 04:06:24.847645 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:24.847611 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr"] Oct 31 04:06:25.810303 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:25.810272 4426 scope.go:122] "RemoveContainer" containerID="59f45e7432347957f62ab10c9ed1b16ffbd56bcd7a3efb41c01e11cad6e2502f" Oct 31 04:06:25.823902 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:25.823867 4426 scope.go:122] "RemoveContainer" containerID="a4de4299a394188b4a6c9bced6fa815ee9910be49fd9b645885204bf42fc1fe3" Oct 31 04:06:30.976374 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:30.976340 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk"] Oct 31 04:06:30.977162 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:30.977138 4426 cpu_manager.go:414] "containerMap: removing container" podUID="16d61453-63b4-4275-97f2-8fd85055f80a" containerName="revoke-keys" Oct 31 04:06:30.977239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:30.977163 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d61453-63b4-4275-97f2-8fd85055f80a" containerName="revoke-keys" Oct 31 04:06:30.977328 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:30.977314 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d61453-63b4-4275-97f2-8fd85055f80a" containerName="revoke-keys" Oct 31 04:06:30.981879 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:30.981862 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:31.031680 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.031641 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:06:31.039722 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.039680 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk"] Oct 31 04:06:31.075965 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.075870 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:06:31.075965 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.075937 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:06:31.075965 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.075939 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:06:31.094857 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.094818 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:31.095055 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.094967 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:31.095429 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.095100 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4w79\" (UniqueName: \"kubernetes.io/projected/64b5cd88-570f-4aa6-995b-a9be1ab07b69-kube-api-access-j4w79\") pod \"maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:31.196430 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.196388 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:31.196619 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.196480 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:31.196695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.196634 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4w79\" (UniqueName: \"kubernetes.io/projected/64b5cd88-570f-4aa6-995b-a9be1ab07b69-kube-api-access-j4w79\") pod \"maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:31.197168 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.197119 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:31.198815 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.198793 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:31.254227 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.254112 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4w79\" (UniqueName: \"kubernetes.io/projected/64b5cd88-570f-4aa6-995b-a9be1ab07b69-kube-api-access-j4w79\") pod \"maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:31.296153 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.296092 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:31.681738 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.681703 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk"] Oct 31 04:06:31.683487 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:06:31.683456 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b5cd88_570f_4aa6_995b_a9be1ab07b69.slice/crio-283f191f74979ac828843bb8c99d1c6870be5fc2e9636adf2c16d6643c5f8dba WatchSource:0}: Error finding container 283f191f74979ac828843bb8c99d1c6870be5fc2e9636adf2c16d6643c5f8dba: Status 404 returned error can't find the container with id 283f191f74979ac828843bb8c99d1c6870be5fc2e9636adf2c16d6643c5f8dba Oct 31 04:06:31.849168 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:31.849118 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" event={"ID":"64b5cd88-570f-4aa6-995b-a9be1ab07b69","Type":"ContainerStarted","Data":"283f191f74979ac828843bb8c99d1c6870be5fc2e9636adf2c16d6643c5f8dba"} Oct 31 04:06:32.855183 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:06:32.855142 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:06:32.863458 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:32.863408 4426 generic.go:356] "Generic (PLEG): container finished" podID="64b5cd88-570f-4aa6-995b-a9be1ab07b69" containerID="69a1fa85b1c0d8fb1ba0b7d5c8cf380d8627da455ba0061fc450f366d748e603" exitCode=0 Oct 31 04:06:32.863667 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:32.863643 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" event={"ID":"64b5cd88-570f-4aa6-995b-a9be1ab07b69","Type":"ContainerDied","Data":"69a1fa85b1c0d8fb1ba0b7d5c8cf380d8627da455ba0061fc450f366d748e603"} Oct 31 04:06:32.867650 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:32.867620 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5"] Oct 31 04:06:32.968545 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:32.968459 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t"] Oct 31 04:06:34.041176 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.041150 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:34.137351 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.137262 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b5cd88-570f-4aa6-995b-a9be1ab07b69-kube-api-access-j4w79\" (UniqueName: \"kubernetes.io/projected/64b5cd88-570f-4aa6-995b-a9be1ab07b69-kube-api-access-j4w79\") pod \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " Oct 31 04:06:34.137351 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.137341 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-xks-ca\") pod \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " Oct 31 04:06:34.137602 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.137418 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-openshift-ca\") pod \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\" (UID: \"64b5cd88-570f-4aa6-995b-a9be1ab07b69\") " Oct 31 04:06:34.137851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.137822 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-openshift-ca" pod "64b5cd88-570f-4aa6-995b-a9be1ab07b69" (UID: "64b5cd88-570f-4aa6-995b-a9be1ab07b69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:34.140087 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.140050 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-xks-ca" pod "64b5cd88-570f-4aa6-995b-a9be1ab07b69" (UID: "64b5cd88-570f-4aa6-995b-a9be1ab07b69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:06:34.140232 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.140106 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b5cd88-570f-4aa6-995b-a9be1ab07b69-kube-api-access-j4w79" pod "64b5cd88-570f-4aa6-995b-a9be1ab07b69" (UID: "64b5cd88-570f-4aa6-995b-a9be1ab07b69"). InnerVolumeSpecName "kube-api-access-j4w79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:34.238768 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.238725 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4w79\" (UniqueName: \"kubernetes.io/projected/64b5cd88-570f-4aa6-995b-a9be1ab07b69-kube-api-access-j4w79\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:34.238768 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.238766 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:34.238768 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.238778 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b5cd88-570f-4aa6-995b-a9be1ab07b69-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:34.883757 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.883722 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" Oct 31 04:06:34.883949 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.883757 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk" event={"ID":"64b5cd88-570f-4aa6-995b-a9be1ab07b69","Type":"ContainerDied","Data":"283f191f74979ac828843bb8c99d1c6870be5fc2e9636adf2c16d6643c5f8dba"} Oct 31 04:06:34.883949 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:34.883840 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="283f191f74979ac828843bb8c99d1c6870be5fc2e9636adf2c16d6643c5f8dba" Oct 31 04:06:35.422372 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:35.422337 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24"] Oct 31 04:06:37.868724 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:37.868682 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" podUID="d05ee558-ced6-41d0-9bff-c01c0862dc4e" containerName="payload-processing" containerID="cri-o://56b5688c45eddd7cbbd8260d000e63f3170c66d854bde4d2f33bbc4067ed55fc" gracePeriod=25 Oct 31 04:06:37.969635 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:37.969582 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" podUID="77082b37-d56f-4c57-aa7d-ed0b60b7b049" containerName="payload-pre-processing" containerID="cri-o://789a73bf2cd3dd4f4c5808ce8aee17ad2957d98f89d229322539cbe2c332640e" gracePeriod=25 Oct 31 04:06:39.823497 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:39.823425 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" containerID="cri-o://1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32" gracePeriod=45 Oct 31 04:06:39.869992 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:39.869946 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" containerID="cri-o://19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0" gracePeriod=45 Oct 31 04:06:39.873516 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:39.873476 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" containerID="cri-o://212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10" gracePeriod=45 Oct 31 04:06:40.186014 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.185987 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 04:06:40.198051 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.198026 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 04:06:40.202974 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.202955 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 04:06:40.331939 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.331888 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-tmp-dir\") pod \"3679293e-d231-4dd6-b1c5-3409a505af6c\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " Oct 31 04:06:40.332332 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.332222 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-tmp-dir" pod "3679293e-d231-4dd6-b1c5-3409a505af6c" (UID: "3679293e-d231-4dd6-b1c5-3409a505af6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.332475 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.332302 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-home\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-home\") pod \"9810df30-5ed6-4aaf-9ec9-e25106217852\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " Oct 31 04:06:40.332632 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.332601 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-home" pod "9810df30-5ed6-4aaf-9ec9-e25106217852" (UID: "9810df30-5ed6-4aaf-9ec9-e25106217852"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.332737 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.332719 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954dfc88-35b6-4f05-b6c3-d083322ff454-tls-certs\" (UniqueName: \"kubernetes.io/secret/954dfc88-35b6-4f05-b6c3-d083322ff454-tls-certs\") pod \"954dfc88-35b6-4f05-b6c3-d083322ff454\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " Oct 31 04:06:40.333465 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.333434 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3679293e-d231-4dd6-b1c5-3409a505af6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3679293e-d231-4dd6-b1c5-3409a505af6c-tls-certs\") pod \"3679293e-d231-4dd6-b1c5-3409a505af6c\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " Oct 31 04:06:40.333580 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.333506 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-tmp-dir\") pod \"954dfc88-35b6-4f05-b6c3-d083322ff454\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " Oct 31 04:06:40.334112 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334084 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-tmp-dir" pod "954dfc88-35b6-4f05-b6c3-d083322ff454" (UID: "954dfc88-35b6-4f05-b6c3-d083322ff454"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.334223 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334181 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-dshm\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-dshm\") pod \"954dfc88-35b6-4f05-b6c3-d083322ff454\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " Oct 31 04:06:40.334302 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334245 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9810df30-5ed6-4aaf-9ec9-e25106217852-tls-certs\" (UniqueName: \"kubernetes.io/secret/9810df30-5ed6-4aaf-9ec9-e25106217852-tls-certs\") pod \"9810df30-5ed6-4aaf-9ec9-e25106217852\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " Oct 31 04:06:40.334621 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334600 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-dshm\") pod \"3679293e-d231-4dd6-b1c5-3409a505af6c\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " Oct 31 04:06:40.334734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334653 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9810df30-5ed6-4aaf-9ec9-e25106217852-kube-api-access-7zmzn\" (UniqueName: \"kubernetes.io/projected/9810df30-5ed6-4aaf-9ec9-e25106217852-kube-api-access-7zmzn\") pod \"9810df30-5ed6-4aaf-9ec9-e25106217852\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " Oct 31 04:06:40.334734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334700 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-model-cache\") pod \"954dfc88-35b6-4f05-b6c3-d083322ff454\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " Oct 31 04:06:40.334866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334741 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-home\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-home\") pod \"3679293e-d231-4dd6-b1c5-3409a505af6c\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " Oct 31 04:06:40.334866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334772 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-home\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-home\") pod \"954dfc88-35b6-4f05-b6c3-d083322ff454\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " Oct 31 04:06:40.334866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334813 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-dshm\") pod \"9810df30-5ed6-4aaf-9ec9-e25106217852\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " Oct 31 04:06:40.335037 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334871 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-model-cache\") pod \"9810df30-5ed6-4aaf-9ec9-e25106217852\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " Oct 31 04:06:40.335037 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334916 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954dfc88-35b6-4f05-b6c3-d083322ff454-kube-api-access-s56l5\" (UniqueName: \"kubernetes.io/projected/954dfc88-35b6-4f05-b6c3-d083322ff454-kube-api-access-s56l5\") pod \"954dfc88-35b6-4f05-b6c3-d083322ff454\" (UID: \"954dfc88-35b6-4f05-b6c3-d083322ff454\") " Oct 31 04:06:40.335037 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334948 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-tmp-dir\") pod \"9810df30-5ed6-4aaf-9ec9-e25106217852\" (UID: \"9810df30-5ed6-4aaf-9ec9-e25106217852\") " Oct 31 04:06:40.335037 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.334994 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-model-cache\") pod \"3679293e-d231-4dd6-b1c5-3409a505af6c\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " Oct 31 04:06:40.335293 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.335054 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3679293e-d231-4dd6-b1c5-3409a505af6c-kube-api-access-2zjkm\" (UniqueName: \"kubernetes.io/projected/3679293e-d231-4dd6-b1c5-3409a505af6c-kube-api-access-2zjkm\") pod \"3679293e-d231-4dd6-b1c5-3409a505af6c\" (UID: \"3679293e-d231-4dd6-b1c5-3409a505af6c\") " Oct 31 04:06:40.335365 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.335342 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-model-cache" pod "954dfc88-35b6-4f05-b6c3-d083322ff454" (UID: "954dfc88-35b6-4f05-b6c3-d083322ff454"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.335761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.335736 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.335863 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.335767 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.335863 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.335781 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.335863 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.335795 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.336694 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.336298 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-home" pod "3679293e-d231-4dd6-b1c5-3409a505af6c" (UID: "3679293e-d231-4dd6-b1c5-3409a505af6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.336694 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.336429 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-home" pod "954dfc88-35b6-4f05-b6c3-d083322ff454" (UID: "954dfc88-35b6-4f05-b6c3-d083322ff454"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.336694 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.336592 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-model-cache" pod "9810df30-5ed6-4aaf-9ec9-e25106217852" (UID: "9810df30-5ed6-4aaf-9ec9-e25106217852"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.336694 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.336654 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-tmp-dir" pod "9810df30-5ed6-4aaf-9ec9-e25106217852" (UID: "9810df30-5ed6-4aaf-9ec9-e25106217852"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.336976 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.336893 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-model-cache" pod "3679293e-d231-4dd6-b1c5-3409a505af6c" (UID: "3679293e-d231-4dd6-b1c5-3409a505af6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.341605 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.341513 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954dfc88-35b6-4f05-b6c3-d083322ff454-kube-api-access-s56l5" pod "954dfc88-35b6-4f05-b6c3-d083322ff454" (UID: "954dfc88-35b6-4f05-b6c3-d083322ff454"). InnerVolumeSpecName "kube-api-access-s56l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:40.343028 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.342993 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-dshm" pod "9810df30-5ed6-4aaf-9ec9-e25106217852" (UID: "9810df30-5ed6-4aaf-9ec9-e25106217852"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.343303 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.343275 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3679293e-d231-4dd6-b1c5-3409a505af6c-kube-api-access-2zjkm" pod "3679293e-d231-4dd6-b1c5-3409a505af6c" (UID: "3679293e-d231-4dd6-b1c5-3409a505af6c"). InnerVolumeSpecName "kube-api-access-2zjkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:40.344596 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.344572 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3679293e-d231-4dd6-b1c5-3409a505af6c-tls-certs" pod "3679293e-d231-4dd6-b1c5-3409a505af6c" (UID: "3679293e-d231-4dd6-b1c5-3409a505af6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:06:40.344596 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.344583 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-dshm" pod "954dfc88-35b6-4f05-b6c3-d083322ff454" (UID: "954dfc88-35b6-4f05-b6c3-d083322ff454"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.344827 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.344659 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954dfc88-35b6-4f05-b6c3-d083322ff454-tls-certs" pod "954dfc88-35b6-4f05-b6c3-d083322ff454" (UID: "954dfc88-35b6-4f05-b6c3-d083322ff454"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:06:40.345363 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.345323 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-dshm" pod "3679293e-d231-4dd6-b1c5-3409a505af6c" (UID: "3679293e-d231-4dd6-b1c5-3409a505af6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:40.345878 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.345840 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9810df30-5ed6-4aaf-9ec9-e25106217852-kube-api-access-7zmzn" pod "9810df30-5ed6-4aaf-9ec9-e25106217852" (UID: "9810df30-5ed6-4aaf-9ec9-e25106217852"). InnerVolumeSpecName "kube-api-access-7zmzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:40.347009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.346981 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9810df30-5ed6-4aaf-9ec9-e25106217852-tls-certs" pod "9810df30-5ed6-4aaf-9ec9-e25106217852" (UID: "9810df30-5ed6-4aaf-9ec9-e25106217852"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:06:40.437273 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437232 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9810df30-5ed6-4aaf-9ec9-e25106217852-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437273 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437268 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437285 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zmzn\" (UniqueName: \"kubernetes.io/projected/9810df30-5ed6-4aaf-9ec9-e25106217852-kube-api-access-7zmzn\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437301 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437315 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437328 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437341 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437354 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s56l5\" (UniqueName: \"kubernetes.io/projected/954dfc88-35b6-4f05-b6c3-d083322ff454-kube-api-access-s56l5\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437368 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9810df30-5ed6-4aaf-9ec9-e25106217852-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437382 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3679293e-d231-4dd6-b1c5-3409a505af6c-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437395 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zjkm\" (UniqueName: \"kubernetes.io/projected/3679293e-d231-4dd6-b1c5-3409a505af6c-kube-api-access-2zjkm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437409 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954dfc88-35b6-4f05-b6c3-d083322ff454-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437421 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3679293e-d231-4dd6-b1c5-3409a505af6c-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.437547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.437434 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954dfc88-35b6-4f05-b6c3-d083322ff454-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:40.939042 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.938998 4426 generic.go:356] "Generic (PLEG): container finished" podID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerID="212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10" exitCode=0 Oct 31 04:06:40.939556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.939054 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" Oct 31 04:06:40.939556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.939094 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" event={"ID":"954dfc88-35b6-4f05-b6c3-d083322ff454","Type":"ContainerDied","Data":"212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10"} Oct 31 04:06:40.939556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.939170 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj" event={"ID":"954dfc88-35b6-4f05-b6c3-d083322ff454","Type":"ContainerDied","Data":"4c27ac417a01cdd852db980b19a0c73c0fe8230407d5838d553983a60a604b26"} Oct 31 04:06:40.939556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.939220 4426 scope.go:122] "RemoveContainer" containerID="212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10" Oct 31 04:06:40.941252 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.941215 4426 generic.go:356] "Generic (PLEG): container finished" podID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerID="19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0" exitCode=0 Oct 31 04:06:40.941368 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.941344 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" event={"ID":"9810df30-5ed6-4aaf-9ec9-e25106217852","Type":"ContainerDied","Data":"19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0"} Oct 31 04:06:40.941433 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.941384 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" event={"ID":"9810df30-5ed6-4aaf-9ec9-e25106217852","Type":"ContainerDied","Data":"aca35e041516798d836b38c6842d8f82447b9b8421ce58fa2f8ca79106ff46ae"} Oct 31 04:06:40.941472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.941451 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr" Oct 31 04:06:40.950328 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.949805 4426 generic.go:356] "Generic (PLEG): container finished" podID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerID="1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32" exitCode=0 Oct 31 04:06:40.950328 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.950022 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" event={"ID":"3679293e-d231-4dd6-b1c5-3409a505af6c","Type":"ContainerDied","Data":"1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32"} Oct 31 04:06:40.950328 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.950057 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" event={"ID":"3679293e-d231-4dd6-b1c5-3409a505af6c","Type":"ContainerDied","Data":"d2858785717976d19628ed297521614750e75dcd65394312295959bd5016669c"} Oct 31 04:06:40.950328 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.950141 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4" Oct 31 04:06:40.954171 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.954150 4426 scope.go:122] "RemoveContainer" containerID="212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10" Oct 31 04:06:40.954491 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:06:40.954461 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10\": container with ID starting with 212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10 not found: ID does not exist" containerID="212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10" Oct 31 04:06:40.954599 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.954501 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10"} err="failed to get container status \"212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10\": rpc error: code = NotFound desc = could not find container \"212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10\": container with ID starting with 212fe7e1791e4824bc4c788053aa4e71c058fcbe84282de73264b95eb59c6a10 not found: ID does not exist" Oct 31 04:06:40.954599 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.954528 4426 scope.go:122] "RemoveContainer" containerID="19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0" Oct 31 04:06:40.973731 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.973706 4426 scope.go:122] "RemoveContainer" containerID="19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0" Oct 31 04:06:40.974074 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:06:40.974053 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0\": container with ID starting with 19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0 not found: ID does not exist" containerID="19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0" Oct 31 04:06:40.974957 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.974088 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0"} err="failed to get container status \"19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0\": rpc error: code = NotFound desc = could not find container \"19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0\": container with ID starting with 19d8ece74d80a0496df4a318bf6cc89c11c03fcfadd4534dd69ec2f659b48fd0 not found: ID does not exist" Oct 31 04:06:40.974957 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.974118 4426 scope.go:122] "RemoveContainer" containerID="1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32" Oct 31 04:06:40.984487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.984464 4426 scope.go:122] "RemoveContainer" containerID="1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32" Oct 31 04:06:40.984790 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:06:40.984766 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32\": container with ID starting with 1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32 not found: ID does not exist" containerID="1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32" Oct 31 04:06:40.984890 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:40.984801 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32"} err="failed to get container status \"1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32\": rpc error: code = NotFound desc = could not find container \"1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32\": container with ID starting with 1963093cf7b11b64b7a1cfe07c1e65fcf04a5008c68a6fda954ca58241b0fc32 not found: ID does not exist" Oct 31 04:06:41.095456 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:41.095419 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj"] Oct 31 04:06:41.144238 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:41.144202 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-w1-d7b7bc-kserve-7b668698fc-p4msj"] Oct 31 04:06:41.201338 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:41.201303 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr"] Oct 31 04:06:41.205272 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:41.205251 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/premium-w1-d7b7bc-kserve-78c6668ffd-72dqr"] Oct 31 04:06:41.306028 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:41.305991 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4"] Oct 31 04:06:41.312480 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:41.312452 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d7b7bc/distinct-2-w1-d7b7bc-kserve-5f5b9c799d-8b5l4"] Oct 31 04:06:42.287238 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:42.287197 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" podUID="77082b37-d56f-4c57-aa7d-ed0b60b7b049" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:06:42.775511 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:42.775479 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws"] Oct 31 04:06:43.140506 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:43.140461 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" path="/var/lib/kubelet/pods/3679293e-d231-4dd6-b1c5-3409a505af6c/volumes" Oct 31 04:06:43.141261 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:43.141236 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" path="/var/lib/kubelet/pods/954dfc88-35b6-4f05-b6c3-d083322ff454/volumes" Oct 31 04:06:43.142116 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:43.142092 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" path="/var/lib/kubelet/pods/9810df30-5ed6-4aaf-9ec9-e25106217852/volumes" Oct 31 04:06:43.294622 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:43.294579 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" podUID="d05ee558-ced6-41d0-9bff-c01c0862dc4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:06:47.776305 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:47.776179 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" podUID="91986eb8-661e-4eb2-8c70-077aa1ed61b8" containerName="payload-processing" containerID="cri-o://fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12" gracePeriod=25 Oct 31 04:06:50.444253 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.444206 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" containerID="cri-o://aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600" gracePeriod=45 Oct 31 04:06:50.711973 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.711900 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 04:06:50.886264 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.886232 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tmp-dir\") pod \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " Oct 31 04:06:50.886447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.886339 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-model-cache\") pod \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " Oct 31 04:06:50.886447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.886430 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-home\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-home\") pod \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " Oct 31 04:06:50.886569 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.886515 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-dshm\") pod \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " Oct 31 04:06:50.886569 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.886551 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tls-certs\") pod \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " Oct 31 04:06:50.886749 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.886546 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tmp-dir" pod "1d00ff31-e8c5-4898-84d8-a4ff023846fe" (UID: "1d00ff31-e8c5-4898-84d8-a4ff023846fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:50.886749 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.886616 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d00ff31-e8c5-4898-84d8-a4ff023846fe-kube-api-access-8vrtm\" (UniqueName: \"kubernetes.io/projected/1d00ff31-e8c5-4898-84d8-a4ff023846fe-kube-api-access-8vrtm\") pod \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\" (UID: \"1d00ff31-e8c5-4898-84d8-a4ff023846fe\") " Oct 31 04:06:50.886749 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.886626 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-model-cache" pod "1d00ff31-e8c5-4898-84d8-a4ff023846fe" (UID: "1d00ff31-e8c5-4898-84d8-a4ff023846fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:50.886749 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.886725 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-home" pod "1d00ff31-e8c5-4898-84d8-a4ff023846fe" (UID: "1d00ff31-e8c5-4898-84d8-a4ff023846fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:50.887103 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.887080 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:50.887216 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.887111 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:50.887216 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.887142 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:50.888970 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.888945 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-dshm" pod "1d00ff31-e8c5-4898-84d8-a4ff023846fe" (UID: "1d00ff31-e8c5-4898-84d8-a4ff023846fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:06:50.888970 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.888955 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tls-certs" pod "1d00ff31-e8c5-4898-84d8-a4ff023846fe" (UID: "1d00ff31-e8c5-4898-84d8-a4ff023846fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:06:50.890281 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.890256 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d00ff31-e8c5-4898-84d8-a4ff023846fe-kube-api-access-8vrtm" pod "1d00ff31-e8c5-4898-84d8-a4ff023846fe" (UID: "1d00ff31-e8c5-4898-84d8-a4ff023846fe"). InnerVolumeSpecName "kube-api-access-8vrtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:50.988163 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.988042 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vrtm\" (UniqueName: \"kubernetes.io/projected/1d00ff31-e8c5-4898-84d8-a4ff023846fe-kube-api-access-8vrtm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:50.988163 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.988082 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d00ff31-e8c5-4898-84d8-a4ff023846fe-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:50.988163 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:50.988092 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d00ff31-e8c5-4898-84d8-a4ff023846fe-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:51.045952 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:51.045912 4426 generic.go:356] "Generic (PLEG): container finished" podID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerID="aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600" exitCode=0 Oct 31 04:06:51.046207 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:51.046160 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" event={"ID":"1d00ff31-e8c5-4898-84d8-a4ff023846fe","Type":"ContainerDied","Data":"aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600"} Oct 31 04:06:51.046308 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:51.046211 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" event={"ID":"1d00ff31-e8c5-4898-84d8-a4ff023846fe","Type":"ContainerDied","Data":"910134076d328ae0647fe04c38c35501df20a5fc1dfb215009356866daaca9e2"} Oct 31 04:06:51.046308 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:51.046262 4426 scope.go:122] "RemoveContainer" containerID="aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600" Oct 31 04:06:51.047558 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:51.047513 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24" Oct 31 04:06:51.062523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:51.061503 4426 scope.go:122] "RemoveContainer" containerID="aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600" Oct 31 04:06:51.062523 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:06:51.061872 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600\": container with ID starting with aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600 not found: ID does not exist" containerID="aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600" Oct 31 04:06:51.062523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:51.061904 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600"} err="failed to get container status \"aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600\": rpc error: code = NotFound desc = could not find container \"aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600\": container with ID starting with aa27fe0e273dc3e434709f6e064240e97fc65b71395b7078b58ad1075887f600 not found: ID does not exist" Oct 31 04:06:51.210703 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:51.210663 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24"] Oct 31 04:06:51.218252 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:51.218213 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-537e34/premium-w3-537e34-kserve-68668dd94-hdf24"] Oct 31 04:06:52.287018 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:52.286959 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" podUID="77082b37-d56f-4c57-aa7d-ed0b60b7b049" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:06:53.133422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:53.133388 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" path="/var/lib/kubelet/pods/1d00ff31-e8c5-4898-84d8-a4ff023846fe/volumes" Oct 31 04:06:53.295079 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:53.295035 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" podUID="d05ee558-ced6-41d0-9bff-c01c0862dc4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:06:54.114092 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:54.114042 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" podUID="91986eb8-661e-4eb2-8c70-077aa1ed61b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:06:58.110298 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.110264 4426 generic.go:356] "Generic (PLEG): container finished" podID="d05ee558-ced6-41d0-9bff-c01c0862dc4e" containerID="56b5688c45eddd7cbbd8260d000e63f3170c66d854bde4d2f33bbc4067ed55fc" exitCode=0 Oct 31 04:06:58.110744 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.110335 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" event={"ID":"d05ee558-ced6-41d0-9bff-c01c0862dc4e","Type":"ContainerDied","Data":"56b5688c45eddd7cbbd8260d000e63f3170c66d854bde4d2f33bbc4067ed55fc"} Oct 31 04:06:58.112761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.112722 4426 generic.go:356] "Generic (PLEG): container finished" podID="77082b37-d56f-4c57-aa7d-ed0b60b7b049" containerID="789a73bf2cd3dd4f4c5808ce8aee17ad2957d98f89d229322539cbe2c332640e" exitCode=0 Oct 31 04:06:58.112932 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.112793 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" event={"ID":"77082b37-d56f-4c57-aa7d-ed0b60b7b049","Type":"ContainerDied","Data":"789a73bf2cd3dd4f4c5808ce8aee17ad2957d98f89d229322539cbe2c332640e"} Oct 31 04:06:58.160797 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.160770 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 04:06:58.185708 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.185671 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05ee558-ced6-41d0-9bff-c01c0862dc4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05ee558-ced6-41d0-9bff-c01c0862dc4e-plugins-config-volume\") pod \"d05ee558-ced6-41d0-9bff-c01c0862dc4e\" (UID: \"d05ee558-ced6-41d0-9bff-c01c0862dc4e\") " Oct 31 04:06:58.185906 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.185842 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05ee558-ced6-41d0-9bff-c01c0862dc4e-kube-api-access-gxrkt\" (UniqueName: \"kubernetes.io/projected/d05ee558-ced6-41d0-9bff-c01c0862dc4e-kube-api-access-gxrkt\") pod \"d05ee558-ced6-41d0-9bff-c01c0862dc4e\" (UID: \"d05ee558-ced6-41d0-9bff-c01c0862dc4e\") " Oct 31 04:06:58.186173 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.186108 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05ee558-ced6-41d0-9bff-c01c0862dc4e-plugins-config-volume" pod "d05ee558-ced6-41d0-9bff-c01c0862dc4e" (UID: "d05ee558-ced6-41d0-9bff-c01c0862dc4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:58.186677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.186652 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05ee558-ced6-41d0-9bff-c01c0862dc4e-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:58.189066 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.189040 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05ee558-ced6-41d0-9bff-c01c0862dc4e-kube-api-access-gxrkt" pod "d05ee558-ced6-41d0-9bff-c01c0862dc4e" (UID: "d05ee558-ced6-41d0-9bff-c01c0862dc4e"). InnerVolumeSpecName "kube-api-access-gxrkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:58.262905 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.262877 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 04:06:58.287545 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.287508 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77082b37-d56f-4c57-aa7d-ed0b60b7b049-kube-api-access-vgs47\" (UniqueName: \"kubernetes.io/projected/77082b37-d56f-4c57-aa7d-ed0b60b7b049-kube-api-access-vgs47\") pod \"77082b37-d56f-4c57-aa7d-ed0b60b7b049\" (UID: \"77082b37-d56f-4c57-aa7d-ed0b60b7b049\") " Oct 31 04:06:58.287819 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.287797 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77082b37-d56f-4c57-aa7d-ed0b60b7b049-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77082b37-d56f-4c57-aa7d-ed0b60b7b049-plugins-config-volume\") pod \"77082b37-d56f-4c57-aa7d-ed0b60b7b049\" (UID: \"77082b37-d56f-4c57-aa7d-ed0b60b7b049\") " Oct 31 04:06:58.288326 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.288289 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77082b37-d56f-4c57-aa7d-ed0b60b7b049-plugins-config-volume" pod "77082b37-d56f-4c57-aa7d-ed0b60b7b049" (UID: "77082b37-d56f-4c57-aa7d-ed0b60b7b049"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:58.288509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.288487 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxrkt\" (UniqueName: \"kubernetes.io/projected/d05ee558-ced6-41d0-9bff-c01c0862dc4e-kube-api-access-gxrkt\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:58.288590 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.288514 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77082b37-d56f-4c57-aa7d-ed0b60b7b049-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:58.289863 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.289840 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77082b37-d56f-4c57-aa7d-ed0b60b7b049-kube-api-access-vgs47" pod "77082b37-d56f-4c57-aa7d-ed0b60b7b049" (UID: "77082b37-d56f-4c57-aa7d-ed0b60b7b049"). InnerVolumeSpecName "kube-api-access-vgs47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:58.389771 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.389741 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgs47\" (UniqueName: \"kubernetes.io/projected/77082b37-d56f-4c57-aa7d-ed0b60b7b049-kube-api-access-vgs47\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:58.832058 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.831962 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv"] Oct 31 04:06:58.832249 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:58.832224 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerName="istio-proxy" containerID="cri-o://5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7" gracePeriod=30 Oct 31 04:06:59.124661 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.124554 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" Oct 31 04:06:59.126560 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.126529 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" Oct 31 04:06:59.133745 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.133701 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5" event={"ID":"d05ee558-ced6-41d0-9bff-c01c0862dc4e","Type":"ContainerDied","Data":"c1805ae3f6f1b9065734e6e22b9e0eae886dfecf59d33097e3a88545d46d7cc4"} Oct 31 04:06:59.133923 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.133777 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t" event={"ID":"77082b37-d56f-4c57-aa7d-ed0b60b7b049","Type":"ContainerDied","Data":"0a84919314d7a58a59859cfbd91626bf8f04338e909d9534af50648c932e2e56"} Oct 31 04:06:59.133923 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.133816 4426 scope.go:122] "RemoveContainer" containerID="56b5688c45eddd7cbbd8260d000e63f3170c66d854bde4d2f33bbc4067ed55fc" Oct 31 04:06:59.153408 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.153386 4426 scope.go:122] "RemoveContainer" containerID="789a73bf2cd3dd4f4c5808ce8aee17ad2957d98f89d229322539cbe2c332640e" Oct 31 04:06:59.280993 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.280945 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t"] Oct 31 04:06:59.285344 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.285313 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d7b7bc-55cb45996c-j8q8t"] Oct 31 04:06:59.390333 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.390294 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5"] Oct 31 04:06:59.396866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.396832 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d7b7bc-69f76c8f95-kkxs5"] Oct 31 04:06:59.539022 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.538983 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs"] Oct 31 04:06:59.539686 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539663 4426 cpu_manager.go:414] "containerMap: removing container" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" Oct 31 04:06:59.539686 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539682 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539703 4426 cpu_manager.go:414] "containerMap: removing container" podUID="77082b37-d56f-4c57-aa7d-ed0b60b7b049" containerName="payload-pre-processing" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539708 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77082b37-d56f-4c57-aa7d-ed0b60b7b049" containerName="payload-pre-processing" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539728 4426 cpu_manager.go:414] "containerMap: removing container" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539732 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539743 4426 cpu_manager.go:414] "containerMap: removing container" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539751 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539762 4426 cpu_manager.go:414] "containerMap: removing container" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539769 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539784 4426 cpu_manager.go:414] "containerMap: removing container" podUID="d05ee558-ced6-41d0-9bff-c01c0862dc4e" containerName="payload-processing" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539789 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05ee558-ced6-41d0-9bff-c01c0862dc4e" containerName="payload-processing" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539809 4426 cpu_manager.go:414] "containerMap: removing container" podUID="64b5cd88-570f-4aa6-995b-a9be1ab07b69" containerName="revoke-keys" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539816 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b5cd88-570f-4aa6-995b-a9be1ab07b69" containerName="revoke-keys" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539909 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="77082b37-d56f-4c57-aa7d-ed0b60b7b049" containerName="payload-pre-processing" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539921 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="3679293e-d231-4dd6-b1c5-3409a505af6c" containerName="main" Oct 31 04:06:59.539916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539927 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="9810df30-5ed6-4aaf-9ec9-e25106217852" containerName="main" Oct 31 04:06:59.540656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539935 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="954dfc88-35b6-4f05-b6c3-d083322ff454" containerName="main" Oct 31 04:06:59.540656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539941 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05ee558-ced6-41d0-9bff-c01c0862dc4e" containerName="payload-processing" Oct 31 04:06:59.540656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539947 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b5cd88-570f-4aa6-995b-a9be1ab07b69" containerName="revoke-keys" Oct 31 04:06:59.540656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.539954 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d00ff31-e8c5-4898-84d8-a4ff023846fe" containerName="main" Oct 31 04:06:59.545014 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.544995 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:06:59.598529 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.598499 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:06:59.598735 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.598655 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:06:59.598735 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.598718 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:06:59.598899 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.598766 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:06:59.602635 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.602605 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs"] Oct 31 04:06:59.604386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.604355 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tffk\" (UniqueName: \"kubernetes.io/projected/73bb3d25-ee33-4991-88ec-9f84ff8de89b-kube-api-access-9tffk\") pod \"maas-api-revoke-sub-b41da457-nj2gs\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:06:59.604564 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.604413 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b41da457-nj2gs\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:06:59.604654 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.604631 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b41da457-nj2gs\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:06:59.626109 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.626072 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz"] Oct 31 04:06:59.630631 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.630600 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:06:59.685810 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.685702 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz"] Oct 31 04:06:59.706465 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.706430 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l92tb\" (UniqueName: \"kubernetes.io/projected/55f04685-9b38-494b-a38d-560e2411be34-kube-api-access-l92tb\") pod \"maas-api-revoke-sub-32bb28c5-6t2lz\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:06:59.706675 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.706487 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tffk\" (UniqueName: \"kubernetes.io/projected/73bb3d25-ee33-4991-88ec-9f84ff8de89b-kube-api-access-9tffk\") pod \"maas-api-revoke-sub-b41da457-nj2gs\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:06:59.706675 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.706550 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f04685-9b38-494b-a38d-560e2411be34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32bb28c5-6t2lz\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:06:59.706675 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.706582 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b41da457-nj2gs\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:06:59.706839 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.706704 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f04685-9b38-494b-a38d-560e2411be34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32bb28c5-6t2lz\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:06:59.706839 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.706754 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b41da457-nj2gs\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:06:59.707468 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.707441 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b41da457-nj2gs\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:06:59.709245 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.709212 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b41da457-nj2gs\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:06:59.764725 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.764695 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tffk\" (UniqueName: \"kubernetes.io/projected/73bb3d25-ee33-4991-88ec-9f84ff8de89b-kube-api-access-9tffk\") pod \"maas-api-revoke-sub-b41da457-nj2gs\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:06:59.780240 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.780206 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf"] Oct 31 04:06:59.784796 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.784772 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:06:59.807875 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.807835 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f04685-9b38-494b-a38d-560e2411be34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32bb28c5-6t2lz\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:06:59.808090 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.807993 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l92tb\" (UniqueName: \"kubernetes.io/projected/55f04685-9b38-494b-a38d-560e2411be34-kube-api-access-l92tb\") pod \"maas-api-revoke-sub-32bb28c5-6t2lz\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:06:59.808090 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.808048 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f04685-9b38-494b-a38d-560e2411be34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32bb28c5-6t2lz\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:06:59.808759 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.808728 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f04685-9b38-494b-a38d-560e2411be34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32bb28c5-6t2lz\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:06:59.810316 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.810292 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f04685-9b38-494b-a38d-560e2411be34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32bb28c5-6t2lz\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:06:59.841652 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.841615 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf"] Oct 31 04:06:59.863373 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.863341 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:06:59.866202 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.866168 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l92tb\" (UniqueName: \"kubernetes.io/projected/55f04685-9b38-494b-a38d-560e2411be34-kube-api-access-l92tb\") pod \"maas-api-revoke-sub-32bb28c5-6t2lz\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:06:59.910436 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.910392 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncjk\" (UniqueName: \"kubernetes.io/projected/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-kube-api-access-6ncjk\") pod \"maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:06:59.910715 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.910556 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:06:59.910813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.910709 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:06:59.955246 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:06:59.953813 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:07:00.015422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:00.012150 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:07:00.015422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:00.012316 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncjk\" (UniqueName: \"kubernetes.io/projected/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-kube-api-access-6ncjk\") pod \"maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:07:00.015422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:00.013307 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:07:00.015422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:00.013899 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:07:00.015422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:00.014627 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:07:00.098309 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:00.098265 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncjk\" (UniqueName: \"kubernetes.io/projected/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-kube-api-access-6ncjk\") pod \"maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:07:00.100859 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:00.100817 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:07:00.267687 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:00.267603 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs"] Oct 31 04:07:00.271742 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:07:00.271711 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bb3d25_ee33_4991_88ec_9f84ff8de89b.slice/crio-3d77e3835a5dabf9743c19e59e412ced12790712552900b7057a847d62813a41 WatchSource:0}: Error finding container 3d77e3835a5dabf9743c19e59e412ced12790712552900b7057a847d62813a41: Status 404 returned error can't find the container with id 3d77e3835a5dabf9743c19e59e412ced12790712552900b7057a847d62813a41 Oct 31 04:07:00.370250 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:00.370211 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz"] Oct 31 04:07:00.423175 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:07:00.417179 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f04685_9b38_494b_a38d_560e2411be34.slice/crio-72add82e54f25eda3b254b778e2ce9ba56ac2648167d125eee604c5074dbe6f3 WatchSource:0}: Error finding container 72add82e54f25eda3b254b778e2ce9ba56ac2648167d125eee604c5074dbe6f3: Status 404 returned error can't find the container with id 72add82e54f25eda3b254b778e2ce9ba56ac2648167d125eee604c5074dbe6f3 Oct 31 04:07:00.489521 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:00.489479 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf"] Oct 31 04:07:00.494017 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:07:00.493985 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7614726_c3ed_48eb_9a1b_c0554cf3fe10.slice/crio-24ec799fefe5ffd063e38b731869dcf184a143fdc26d8f6e3632705f3812ccb4 WatchSource:0}: Error finding container 24ec799fefe5ffd063e38b731869dcf184a143fdc26d8f6e3632705f3812ccb4: Status 404 returned error can't find the container with id 24ec799fefe5ffd063e38b731869dcf184a143fdc26d8f6e3632705f3812ccb4 Oct 31 04:07:01.137233 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:01.137090 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77082b37-d56f-4c57-aa7d-ed0b60b7b049" path="/var/lib/kubelet/pods/77082b37-d56f-4c57-aa7d-ed0b60b7b049/volumes" Oct 31 04:07:01.137801 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:01.137778 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05ee558-ced6-41d0-9bff-c01c0862dc4e" path="/var/lib/kubelet/pods/d05ee558-ced6-41d0-9bff-c01c0862dc4e/volumes" Oct 31 04:07:01.158068 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:01.158026 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" event={"ID":"b7614726-c3ed-48eb-9a1b-c0554cf3fe10","Type":"ContainerStarted","Data":"e614424eefa3ade289f5a098f97749e6c3928fce0b34cdfd06ed47888d849654"} Oct 31 04:07:01.158357 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:01.158099 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" event={"ID":"b7614726-c3ed-48eb-9a1b-c0554cf3fe10","Type":"ContainerStarted","Data":"24ec799fefe5ffd063e38b731869dcf184a143fdc26d8f6e3632705f3812ccb4"} Oct 31 04:07:01.160869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:01.160832 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" event={"ID":"73bb3d25-ee33-4991-88ec-9f84ff8de89b","Type":"ContainerStarted","Data":"fb60859411416a940587b051518b62d63a24d3a4c275ba3f150a8f2b2ea46412"} Oct 31 04:07:01.161036 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:01.160896 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" event={"ID":"73bb3d25-ee33-4991-88ec-9f84ff8de89b","Type":"ContainerStarted","Data":"3d77e3835a5dabf9743c19e59e412ced12790712552900b7057a847d62813a41"} Oct 31 04:07:01.162988 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:01.162964 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" event={"ID":"55f04685-9b38-494b-a38d-560e2411be34","Type":"ContainerStarted","Data":"261b682ccd4de07c782bbd656f68e682181308392e1db85f6c8bdc2eedae1bb9"} Oct 31 04:07:01.163090 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:01.163005 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" event={"ID":"55f04685-9b38-494b-a38d-560e2411be34","Type":"ContainerStarted","Data":"72add82e54f25eda3b254b778e2ce9ba56ac2648167d125eee604c5074dbe6f3"} Oct 31 04:07:01.270320 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:01.270255 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" podStartSLOduration=2.27023677 podStartE2EDuration="2.27023677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:07:01.268015288 +0000 UTC m=+2318.268640515" watchObservedRunningTime="2026-10-31 04:07:01.27023677 +0000 UTC m=+2318.270861997" Oct 31 04:07:01.377513 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:01.377450 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" podStartSLOduration=2.377434029 podStartE2EDuration="2.377434029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:07:01.376384225 +0000 UTC m=+2318.377009454" watchObservedRunningTime="2026-10-31 04:07:01.377434029 +0000 UTC m=+2318.378059257" Oct 31 04:07:01.486734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:01.486675 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" podStartSLOduration=2.486655303 podStartE2EDuration="2.486655303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:07:01.485191114 +0000 UTC m=+2318.485816343" watchObservedRunningTime="2026-10-31 04:07:01.486655303 +0000 UTC m=+2318.487280534" Oct 31 04:07:02.187196 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:02.184932 4426 generic.go:356] "Generic (PLEG): container finished" podID="b7614726-c3ed-48eb-9a1b-c0554cf3fe10" containerID="e614424eefa3ade289f5a098f97749e6c3928fce0b34cdfd06ed47888d849654" exitCode=0 Oct 31 04:07:02.187196 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:02.185001 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" event={"ID":"b7614726-c3ed-48eb-9a1b-c0554cf3fe10","Type":"ContainerDied","Data":"e614424eefa3ade289f5a098f97749e6c3928fce0b34cdfd06ed47888d849654"} Oct 31 04:07:02.187196 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:02.187075 4426 generic.go:356] "Generic (PLEG): container finished" podID="73bb3d25-ee33-4991-88ec-9f84ff8de89b" containerID="fb60859411416a940587b051518b62d63a24d3a4c275ba3f150a8f2b2ea46412" exitCode=0 Oct 31 04:07:02.187196 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:02.187157 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" event={"ID":"73bb3d25-ee33-4991-88ec-9f84ff8de89b","Type":"ContainerDied","Data":"fb60859411416a940587b051518b62d63a24d3a4c275ba3f150a8f2b2ea46412"} Oct 31 04:07:02.189424 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:02.189390 4426 generic.go:356] "Generic (PLEG): container finished" podID="55f04685-9b38-494b-a38d-560e2411be34" containerID="261b682ccd4de07c782bbd656f68e682181308392e1db85f6c8bdc2eedae1bb9" exitCode=0 Oct 31 04:07:02.189603 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:02.189466 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" event={"ID":"55f04685-9b38-494b-a38d-560e2411be34","Type":"ContainerDied","Data":"261b682ccd4de07c782bbd656f68e682181308392e1db85f6c8bdc2eedae1bb9"} Oct 31 04:07:02.851299 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:07:02.851256 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:07:03.013759 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.013722 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:07:03.013944 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.013788 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:07:03.388673 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.388637 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:07:03.488213 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.488169 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:07:03.491708 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.491684 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:07:03.494019 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.493998 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-xks-ca\") pod \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " Oct 31 04:07:03.494101 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.494047 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-kube-api-access-6ncjk\" (UniqueName: \"kubernetes.io/projected/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-kube-api-access-6ncjk\") pod \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " Oct 31 04:07:03.494224 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.494212 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-openshift-ca\") pod \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\" (UID: \"b7614726-c3ed-48eb-9a1b-c0554cf3fe10\") " Oct 31 04:07:03.494568 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.494539 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-openshift-ca" pod "b7614726-c3ed-48eb-9a1b-c0554cf3fe10" (UID: "b7614726-c3ed-48eb-9a1b-c0554cf3fe10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:03.496251 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.496221 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-xks-ca" pod "b7614726-c3ed-48eb-9a1b-c0554cf3fe10" (UID: "b7614726-c3ed-48eb-9a1b-c0554cf3fe10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:03.496491 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.496470 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-kube-api-access-6ncjk" pod "b7614726-c3ed-48eb-9a1b-c0554cf3fe10" (UID: "b7614726-c3ed-48eb-9a1b-c0554cf3fe10"). InnerVolumeSpecName "kube-api-access-6ncjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:03.595626 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.595584 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f04685-9b38-494b-a38d-560e2411be34-kube-api-access-l92tb\" (UniqueName: \"kubernetes.io/projected/55f04685-9b38-494b-a38d-560e2411be34-kube-api-access-l92tb\") pod \"55f04685-9b38-494b-a38d-560e2411be34\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " Oct 31 04:07:03.595842 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.595689 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-xks-ca\") pod \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " Oct 31 04:07:03.595842 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.595745 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bb3d25-ee33-4991-88ec-9f84ff8de89b-kube-api-access-9tffk\" (UniqueName: \"kubernetes.io/projected/73bb3d25-ee33-4991-88ec-9f84ff8de89b-kube-api-access-9tffk\") pod \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " Oct 31 04:07:03.595842 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.595788 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f04685-9b38-494b-a38d-560e2411be34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f04685-9b38-494b-a38d-560e2411be34-maas-api-xks-ca\") pod \"55f04685-9b38-494b-a38d-560e2411be34\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " Oct 31 04:07:03.595842 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.595823 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f04685-9b38-494b-a38d-560e2411be34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f04685-9b38-494b-a38d-560e2411be34-maas-api-openshift-ca\") pod \"55f04685-9b38-494b-a38d-560e2411be34\" (UID: \"55f04685-9b38-494b-a38d-560e2411be34\") " Oct 31 04:07:03.596073 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.595882 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-openshift-ca\") pod \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\" (UID: \"73bb3d25-ee33-4991-88ec-9f84ff8de89b\") " Oct 31 04:07:03.596381 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.596345 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-openshift-ca" pod "73bb3d25-ee33-4991-88ec-9f84ff8de89b" (UID: "73bb3d25-ee33-4991-88ec-9f84ff8de89b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:03.596496 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.596349 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f04685-9b38-494b-a38d-560e2411be34-maas-api-openshift-ca" pod "55f04685-9b38-494b-a38d-560e2411be34" (UID: "55f04685-9b38-494b-a38d-560e2411be34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:03.597078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.597050 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:03.597209 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.597084 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:03.597209 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.597105 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ncjk\" (UniqueName: \"kubernetes.io/projected/b7614726-c3ed-48eb-9a1b-c0554cf3fe10-kube-api-access-6ncjk\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:03.597209 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.597120 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f04685-9b38-494b-a38d-560e2411be34-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:03.597209 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.597153 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:03.598048 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.598023 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f04685-9b38-494b-a38d-560e2411be34-maas-api-xks-ca" pod "55f04685-9b38-494b-a38d-560e2411be34" (UID: "55f04685-9b38-494b-a38d-560e2411be34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:03.598160 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.598061 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f04685-9b38-494b-a38d-560e2411be34-kube-api-access-l92tb" pod "55f04685-9b38-494b-a38d-560e2411be34" (UID: "55f04685-9b38-494b-a38d-560e2411be34"). InnerVolumeSpecName "kube-api-access-l92tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:03.598160 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.598069 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-xks-ca" pod "73bb3d25-ee33-4991-88ec-9f84ff8de89b" (UID: "73bb3d25-ee33-4991-88ec-9f84ff8de89b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:03.598252 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.598231 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bb3d25-ee33-4991-88ec-9f84ff8de89b-kube-api-access-9tffk" pod "73bb3d25-ee33-4991-88ec-9f84ff8de89b" (UID: "73bb3d25-ee33-4991-88ec-9f84ff8de89b"). InnerVolumeSpecName "kube-api-access-9tffk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:03.698526 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.698478 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tffk\" (UniqueName: \"kubernetes.io/projected/73bb3d25-ee33-4991-88ec-9f84ff8de89b-kube-api-access-9tffk\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:03.698526 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.698519 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f04685-9b38-494b-a38d-560e2411be34-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:03.698526 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.698530 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l92tb\" (UniqueName: \"kubernetes.io/projected/55f04685-9b38-494b-a38d-560e2411be34-kube-api-access-l92tb\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:03.698526 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:03.698540 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bb3d25-ee33-4991-88ec-9f84ff8de89b-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:04.094292 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.094258 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 04:07:04.114446 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.114346 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" podUID="91986eb8-661e-4eb2-8c70-077aa1ed61b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:07:04.203909 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.203870 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-podinfo\") pod \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " Oct 31 04:07:04.204096 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.203946 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-data\") pod \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " Oct 31 04:07:04.204096 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204005 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-token\" (UniqueName: \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-token\") pod \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " Oct 31 04:07:04.204096 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204078 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-envoy\") pod \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " Oct 31 04:07:04.204358 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204136 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-certs\") pod \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " Oct 31 04:07:04.204358 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204211 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istiod-ca-cert\") pod \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " Oct 31 04:07:04.204358 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204294 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-credential-socket\") pod \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " Oct 31 04:07:04.204358 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204352 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-kube-api-access-mgkwb\" (UniqueName: \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-kube-api-access-mgkwb\") pod \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " Oct 31 04:07:04.204578 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204407 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-socket\") pod \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\" (UID: \"fa5412c3-19d3-4d81-b59a-20c482bc8b75\") " Oct 31 04:07:04.204578 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204439 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-data" pod "fa5412c3-19d3-4d81-b59a-20c482bc8b75" (UID: "fa5412c3-19d3-4d81-b59a-20c482bc8b75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:04.204578 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204457 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-certs" pod "fa5412c3-19d3-4d81-b59a-20c482bc8b75" (UID: "fa5412c3-19d3-4d81-b59a-20c482bc8b75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:04.204738 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204639 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-credential-socket" pod "fa5412c3-19d3-4d81-b59a-20c482bc8b75" (UID: "fa5412c3-19d3-4d81-b59a-20c482bc8b75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:04.204800 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204732 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istiod-ca-cert" pod "fa5412c3-19d3-4d81-b59a-20c482bc8b75" (UID: "fa5412c3-19d3-4d81-b59a-20c482bc8b75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:04.204867 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.204849 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-socket" pod "fa5412c3-19d3-4d81-b59a-20c482bc8b75" (UID: "fa5412c3-19d3-4d81-b59a-20c482bc8b75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:04.205065 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.205036 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:04.205065 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.205067 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:04.205305 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.205083 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:04.205305 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.205099 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:04.205305 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.205115 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:04.206643 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.206614 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-podinfo" pod "fa5412c3-19d3-4d81-b59a-20c482bc8b75" (UID: "fa5412c3-19d3-4d81-b59a-20c482bc8b75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:07:04.206733 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.206676 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-envoy" pod "fa5412c3-19d3-4d81-b59a-20c482bc8b75" (UID: "fa5412c3-19d3-4d81-b59a-20c482bc8b75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:04.206862 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.206835 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-token" pod "fa5412c3-19d3-4d81-b59a-20c482bc8b75" (UID: "fa5412c3-19d3-4d81-b59a-20c482bc8b75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:04.206972 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.206925 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-kube-api-access-mgkwb" pod "fa5412c3-19d3-4d81-b59a-20c482bc8b75" (UID: "fa5412c3-19d3-4d81-b59a-20c482bc8b75"). InnerVolumeSpecName "kube-api-access-mgkwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:04.208953 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.208929 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" Oct 31 04:07:04.209081 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.209013 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz" event={"ID":"55f04685-9b38-494b-a38d-560e2411be34","Type":"ContainerDied","Data":"72add82e54f25eda3b254b778e2ce9ba56ac2648167d125eee604c5074dbe6f3"} Oct 31 04:07:04.209081 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.209066 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72add82e54f25eda3b254b778e2ce9ba56ac2648167d125eee604c5074dbe6f3" Oct 31 04:07:04.211090 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.211059 4426 generic.go:356] "Generic (PLEG): container finished" podID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerID="5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7" exitCode=0 Oct 31 04:07:04.211265 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.211148 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" Oct 31 04:07:04.211336 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.211287 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" event={"ID":"fa5412c3-19d3-4d81-b59a-20c482bc8b75","Type":"ContainerDied","Data":"5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7"} Oct 31 04:07:04.211395 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.211336 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv" event={"ID":"fa5412c3-19d3-4d81-b59a-20c482bc8b75","Type":"ContainerDied","Data":"a2374dc7f3ea428eb6465c3e81e54f7952040c479f31c3617eb71d910da19fb4"} Oct 31 04:07:04.211395 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.211358 4426 scope.go:122] "RemoveContainer" containerID="5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7" Oct 31 04:07:04.216217 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.216134 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" event={"ID":"b7614726-c3ed-48eb-9a1b-c0554cf3fe10","Type":"ContainerDied","Data":"24ec799fefe5ffd063e38b731869dcf184a143fdc26d8f6e3632705f3812ccb4"} Oct 31 04:07:04.216217 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.216174 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf" Oct 31 04:07:04.216217 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.216193 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24ec799fefe5ffd063e38b731869dcf184a143fdc26d8f6e3632705f3812ccb4" Oct 31 04:07:04.224471 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.224274 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt"] Oct 31 04:07:04.242180 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.233681 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" event={"ID":"73bb3d25-ee33-4991-88ec-9f84ff8de89b","Type":"ContainerDied","Data":"3d77e3835a5dabf9743c19e59e412ced12790712552900b7057a847d62813a41"} Oct 31 04:07:04.242180 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.233735 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d77e3835a5dabf9743c19e59e412ced12790712552900b7057a847d62813a41" Oct 31 04:07:04.242180 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.233856 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs" Oct 31 04:07:04.268168 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.261315 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2"] Oct 31 04:07:04.268168 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.266166 4426 scope.go:122] "RemoveContainer" containerID="5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7" Oct 31 04:07:04.268168 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:07:04.266727 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7\": container with ID starting with 5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7 not found: ID does not exist" containerID="5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7" Oct 31 04:07:04.268168 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.266766 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7"} err="failed to get container status \"5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7\": rpc error: code = NotFound desc = could not find container \"5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7\": container with ID starting with 5b40ce1b3f2e1765ecfa8242d2cd65323431a4b3486a6ae7a6923182235539d7 not found: ID does not exist" Oct 31 04:07:04.269637 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.269608 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5"] Oct 31 04:07:04.305912 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.305876 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgkwb\" (UniqueName: \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-kube-api-access-mgkwb\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:04.305912 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.305913 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:04.306178 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.305931 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:04.306178 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.305957 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5412c3-19d3-4d81-b59a-20c482bc8b75-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:04.762453 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.762412 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv"] Oct 31 04:07:04.768744 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:04.768707 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-txrkv"] Oct 31 04:07:05.136518 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:05.136420 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" path="/var/lib/kubelet/pods/fa5412c3-19d3-4d81-b59a-20c482bc8b75/volumes" Oct 31 04:07:08.087968 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.087933 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 04:07:08.156676 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.156640 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91986eb8-661e-4eb2-8c70-077aa1ed61b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91986eb8-661e-4eb2-8c70-077aa1ed61b8-plugins-config-volume\") pod \"91986eb8-661e-4eb2-8c70-077aa1ed61b8\" (UID: \"91986eb8-661e-4eb2-8c70-077aa1ed61b8\") " Oct 31 04:07:08.156962 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.156935 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91986eb8-661e-4eb2-8c70-077aa1ed61b8-kube-api-access-jjnmn\" (UniqueName: \"kubernetes.io/projected/91986eb8-661e-4eb2-8c70-077aa1ed61b8-kube-api-access-jjnmn\") pod \"91986eb8-661e-4eb2-8c70-077aa1ed61b8\" (UID: \"91986eb8-661e-4eb2-8c70-077aa1ed61b8\") " Oct 31 04:07:08.157178 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.157115 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91986eb8-661e-4eb2-8c70-077aa1ed61b8-plugins-config-volume" pod "91986eb8-661e-4eb2-8c70-077aa1ed61b8" (UID: "91986eb8-661e-4eb2-8c70-077aa1ed61b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:08.157450 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.157419 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91986eb8-661e-4eb2-8c70-077aa1ed61b8-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:08.159289 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.159260 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91986eb8-661e-4eb2-8c70-077aa1ed61b8-kube-api-access-jjnmn" pod "91986eb8-661e-4eb2-8c70-077aa1ed61b8" (UID: "91986eb8-661e-4eb2-8c70-077aa1ed61b8"). InnerVolumeSpecName "kube-api-access-jjnmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:08.258824 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.258567 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjnmn\" (UniqueName: \"kubernetes.io/projected/91986eb8-661e-4eb2-8c70-077aa1ed61b8-kube-api-access-jjnmn\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:08.279951 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.279917 4426 generic.go:356] "Generic (PLEG): container finished" podID="91986eb8-661e-4eb2-8c70-077aa1ed61b8" containerID="fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12" exitCode=0 Oct 31 04:07:08.280148 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.279981 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" Oct 31 04:07:08.280148 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.279997 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" event={"ID":"91986eb8-661e-4eb2-8c70-077aa1ed61b8","Type":"ContainerDied","Data":"fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12"} Oct 31 04:07:08.280148 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.280050 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws" event={"ID":"91986eb8-661e-4eb2-8c70-077aa1ed61b8","Type":"ContainerDied","Data":"d64c68df14b7b44d7ba03e88053b90e054a707a1298ac1196130961cded272e9"} Oct 31 04:07:08.280148 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.280072 4426 scope.go:122] "RemoveContainer" containerID="fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12" Oct 31 04:07:08.291758 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.291732 4426 scope.go:122] "RemoveContainer" containerID="fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12" Oct 31 04:07:08.292111 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:07:08.292087 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12\": container with ID starting with fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12 not found: ID does not exist" containerID="fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12" Oct 31 04:07:08.292173 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.292151 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12"} err="failed to get container status \"fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12\": rpc error: code = NotFound desc = could not find container \"fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12\": container with ID starting with fe459045ad7915dc294adc80fcb29019cfe28fa10c229d2a08dc7685ff765b12 not found: ID does not exist" Oct 31 04:07:08.364253 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.364111 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:07:08.437306 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.437267 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws"] Oct 31 04:07:08.441404 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:08.441377 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-537e34-546b788fcf-644ws"] Oct 31 04:07:09.130191 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:09.130156 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91986eb8-661e-4eb2-8c70-077aa1ed61b8" path="/var/lib/kubelet/pods/91986eb8-661e-4eb2-8c70-077aa1ed61b8/volumes" Oct 31 04:07:11.385461 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:11.385424 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq"] Oct 31 04:07:13.846587 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:13.846543 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52"] Oct 31 04:07:13.847043 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:13.846820 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" containerID="cri-o://7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353" gracePeriod=30 Oct 31 04:07:16.386528 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:16.386469 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" podUID="89cd7dd5-9af7-40f3-b01f-96eb4f64aae6" containerName="payload-processing" containerID="cri-o://c809e023f856a11cecbb5885eaa9015a97e117d06e93bd20fa26e94ad36541c8" gracePeriod=25 Oct 31 04:07:17.307420 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:17.307380 4426 patch_prober.go:29] interesting pod/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:07:17.307595 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:17.307441 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:07:17.326113 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:17.326074 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" podUID="89cd7dd5-9af7-40f3-b01f-96eb4f64aae6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:07:19.130806 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.130716 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 04:07:19.220329 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220294 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-certs\") pod \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " Oct 31 04:07:19.220329 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220340 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-envoy\") pod \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " Oct 31 04:07:19.220607 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220365 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-podinfo\") pod \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " Oct 31 04:07:19.220607 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220467 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-credential-socket\") pod \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " Oct 31 04:07:19.220607 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220510 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istiod-ca-cert\") pod \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " Oct 31 04:07:19.220607 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220571 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-socket\") pod \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " Oct 31 04:07:19.220840 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220614 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-kube-api-access-47hhg\" (UniqueName: \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-kube-api-access-47hhg\") pod \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " Oct 31 04:07:19.220840 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220670 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-token\" (UniqueName: \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-token\") pod \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " Oct 31 04:07:19.220840 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220693 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-certs" pod "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" (UID: "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.220840 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220804 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-data\") pod \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\" (UID: \"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b\") " Oct 31 04:07:19.221539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220842 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-credential-socket" pod "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" (UID: "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.221539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220945 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-socket" pod "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" (UID: "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.221539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.220997 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istiod-ca-cert" pod "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" (UID: "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:19.221539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.221227 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-data" pod "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" (UID: "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.221539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.221472 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.221539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.221493 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.221539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.221511 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.221539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.221526 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.221539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.221539 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.222834 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.222796 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-podinfo" pod "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" (UID: "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:07:19.222953 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.222853 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-envoy" pod "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" (UID: "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.223142 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.223106 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-token" pod "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" (UID: "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:19.223258 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.223238 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-kube-api-access-47hhg" pod "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" (UID: "d5462e40-4af4-4dc9-9a84-4169c3e9fd3b"). InnerVolumeSpecName "kube-api-access-47hhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:19.263635 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.263572 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" containerID="cri-o://f33cf7346e68bfe4c8345ed244f29f129f07bc13ffa712be663300fd90b75f77" gracePeriod=45 Oct 31 04:07:19.285036 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.284950 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" containerID="cri-o://7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e" gracePeriod=45 Oct 31 04:07:19.292869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.292827 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" containerID="cri-o://bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360" gracePeriod=45 Oct 31 04:07:19.322334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.322299 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.322334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.322333 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.322544 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.322350 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47hhg\" (UniqueName: \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-kube-api-access-47hhg\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.322544 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.322361 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.382680 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:07:19.382567 4426 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f5f78d0_b661_4846_a078_e80aefc88592.slice/crio-conmon-7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod638fac5f_cc53_4bb2_b6c8_6fd2f0ace35c.slice/crio-bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod638fac5f_cc53_4bb2_b6c8_6fd2f0ace35c.slice/crio-conmon-bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360.scope\": RecentStats: unable to find data in memory cache]" Oct 31 04:07:19.399045 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.399004 4426 generic.go:356] "Generic (PLEG): container finished" podID="dcb7866b-4998-420e-9afb-29ce3903955c" containerID="f33cf7346e68bfe4c8345ed244f29f129f07bc13ffa712be663300fd90b75f77" exitCode=0 Oct 31 04:07:19.399199 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.399039 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" event={"ID":"dcb7866b-4998-420e-9afb-29ce3903955c","Type":"ContainerDied","Data":"f33cf7346e68bfe4c8345ed244f29f129f07bc13ffa712be663300fd90b75f77"} Oct 31 04:07:19.401237 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.401204 4426 generic.go:356] "Generic (PLEG): container finished" podID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerID="7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353" exitCode=0 Oct 31 04:07:19.401385 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.401272 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" Oct 31 04:07:19.401385 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.401284 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" event={"ID":"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b","Type":"ContainerDied","Data":"7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353"} Oct 31 04:07:19.401543 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.401420 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52" event={"ID":"d5462e40-4af4-4dc9-9a84-4169c3e9fd3b","Type":"ContainerDied","Data":"b51bd8a768606007e8dae62973479437d2b81a65fd8b14fc2e74478c8ccb9f8e"} Oct 31 04:07:19.401543 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.401446 4426 scope.go:122] "RemoveContainer" containerID="7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353" Oct 31 04:07:19.404843 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.404802 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": dial tcp 10.133.0.86:8000: connect: connection refused" Oct 31 04:07:19.405517 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.405485 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": dial tcp 10.133.0.79:8000: connect: connection refused" Oct 31 04:07:19.416993 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.416958 4426 scope.go:122] "RemoveContainer" containerID="7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353" Oct 31 04:07:19.417468 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:07:19.417443 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353\": container with ID starting with 7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353 not found: ID does not exist" containerID="7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353" Oct 31 04:07:19.417556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.417481 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353"} err="failed to get container status \"7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353\": rpc error: code = NotFound desc = could not find container \"7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353\": container with ID starting with 7e8628fc6c05786e787391c8c7d797867fff0cf4605e31b0aae0a3cc7a2d9353 not found: ID does not exist" Oct 31 04:07:19.418808 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.418780 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": dial tcp 10.133.0.118:8000: connect: connection refused" Oct 31 04:07:19.588159 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.585061 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52"] Oct 31 04:07:19.590312 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.590261 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae8e27-openshift-default-c87fdc98f-d4v52"] Oct 31 04:07:19.612008 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.611982 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 04:07:19.616604 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.616582 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 04:07:19.643627 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.643602 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 04:07:19.749360 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.749324 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-dshm\") pod \"dcb7866b-4998-420e-9afb-29ce3903955c\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " Oct 31 04:07:19.749520 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.749396 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-home\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-home\") pod \"3f5f78d0-b661-4846-a078-e80aefc88592\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " Oct 31 04:07:19.749520 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.749430 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-dshm\") pod \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " Oct 31 04:07:19.749520 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.749464 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-tmp-dir\") pod \"dcb7866b-4998-420e-9afb-29ce3903955c\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " Oct 31 04:07:19.749520 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.749504 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-kube-api-access-vj9zc\" (UniqueName: \"kubernetes.io/projected/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-kube-api-access-vj9zc\") pod \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " Oct 31 04:07:19.750060 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.749748 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-home\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-home\") pod \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " Oct 31 04:07:19.750060 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.749798 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5f78d0-b661-4846-a078-e80aefc88592-kube-api-access-hlwsr\" (UniqueName: \"kubernetes.io/projected/3f5f78d0-b661-4846-a078-e80aefc88592-kube-api-access-hlwsr\") pod \"3f5f78d0-b661-4846-a078-e80aefc88592\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " Oct 31 04:07:19.750060 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.749874 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-model-cache\") pod \"3f5f78d0-b661-4846-a078-e80aefc88592\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " Oct 31 04:07:19.750060 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.749902 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-model-cache\") pod \"dcb7866b-4998-420e-9afb-29ce3903955c\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.750479 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb7866b-4998-420e-9afb-29ce3903955c-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb7866b-4998-420e-9afb-29ce3903955c-tls-certs\") pod \"dcb7866b-4998-420e-9afb-29ce3903955c\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.750519 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-tmp-dir\") pod \"3f5f78d0-b661-4846-a078-e80aefc88592\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.750571 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f5f78d0-b661-4846-a078-e80aefc88592-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5f78d0-b661-4846-a078-e80aefc88592-tls-certs\") pod \"3f5f78d0-b661-4846-a078-e80aefc88592\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.750643 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-home\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-home\") pod \"dcb7866b-4998-420e-9afb-29ce3903955c\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.750724 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tls-certs\" (UniqueName: \"kubernetes.io/secret/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tls-certs\") pod \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.750767 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tmp-dir\") pod \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.750861 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb7866b-4998-420e-9afb-29ce3903955c-kube-api-access-2q6wg\" (UniqueName: \"kubernetes.io/projected/dcb7866b-4998-420e-9afb-29ce3903955c-kube-api-access-2q6wg\") pod \"dcb7866b-4998-420e-9afb-29ce3903955c\" (UID: \"dcb7866b-4998-420e-9afb-29ce3903955c\") " Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.750910 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-dshm\") pod \"3f5f78d0-b661-4846-a078-e80aefc88592\" (UID: \"3f5f78d0-b661-4846-a078-e80aefc88592\") " Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.750941 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-model-cache\") pod \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\" (UID: \"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c\") " Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.750331 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-home" pod "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" (UID: "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.750517 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-tmp-dir" pod "dcb7866b-4998-420e-9afb-29ce3903955c" (UID: "dcb7866b-4998-420e-9afb-29ce3903955c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.751250 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-home" pod "3f5f78d0-b661-4846-a078-e80aefc88592" (UID: "3f5f78d0-b661-4846-a078-e80aefc88592"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.751789 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-home" pod "dcb7866b-4998-420e-9afb-29ce3903955c" (UID: "dcb7866b-4998-420e-9afb-29ce3903955c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.751890 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tmp-dir" pod "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" (UID: "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.752114 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-model-cache" pod "dcb7866b-4998-420e-9afb-29ce3903955c" (UID: "dcb7866b-4998-420e-9afb-29ce3903955c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.752334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.752163 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-model-cache" pod "3f5f78d0-b661-4846-a078-e80aefc88592" (UID: "3f5f78d0-b661-4846-a078-e80aefc88592"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.752369 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-tmp-dir" pod "3f5f78d0-b661-4846-a078-e80aefc88592" (UID: "3f5f78d0-b661-4846-a078-e80aefc88592"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.753199 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-model-cache" pod "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" (UID: "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.753363 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-dshm" pod "dcb7866b-4998-420e-9afb-29ce3903955c" (UID: "dcb7866b-4998-420e-9afb-29ce3903955c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.755516 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.755554 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.755592 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.755605 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.755632 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.755645 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.755659 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.755672 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcb7866b-4998-420e-9afb-29ce3903955c-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.755684 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.755709 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.755963 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-kube-api-access-vj9zc" pod "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" (UID: "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c"). InnerVolumeSpecName "kube-api-access-vj9zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:19.757277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.756736 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-dshm" pod "3f5f78d0-b661-4846-a078-e80aefc88592" (UID: "3f5f78d0-b661-4846-a078-e80aefc88592"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.757982 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.757513 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5f78d0-b661-4846-a078-e80aefc88592-kube-api-access-hlwsr" pod "3f5f78d0-b661-4846-a078-e80aefc88592" (UID: "3f5f78d0-b661-4846-a078-e80aefc88592"). InnerVolumeSpecName "kube-api-access-hlwsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:19.758405 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.758365 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5f78d0-b661-4846-a078-e80aefc88592-tls-certs" pod "3f5f78d0-b661-4846-a078-e80aefc88592" (UID: "3f5f78d0-b661-4846-a078-e80aefc88592"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:19.758405 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.758379 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tls-certs" pod "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" (UID: "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:19.758542 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.758472 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-dshm" pod "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" (UID: "638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:19.758653 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.758610 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb7866b-4998-420e-9afb-29ce3903955c-tls-certs" pod "dcb7866b-4998-420e-9afb-29ce3903955c" (UID: "dcb7866b-4998-420e-9afb-29ce3903955c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:19.759961 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.759939 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb7866b-4998-420e-9afb-29ce3903955c-kube-api-access-2q6wg" pod "dcb7866b-4998-420e-9afb-29ce3903955c" (UID: "dcb7866b-4998-420e-9afb-29ce3903955c"). InnerVolumeSpecName "kube-api-access-2q6wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:19.856873 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.856833 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.856873 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.856873 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q6wg\" (UniqueName: \"kubernetes.io/projected/dcb7866b-4998-420e-9afb-29ce3903955c-kube-api-access-2q6wg\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.857068 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.856891 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5f78d0-b661-4846-a078-e80aefc88592-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.857068 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.856906 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.857068 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.856919 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj9zc\" (UniqueName: \"kubernetes.io/projected/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c-kube-api-access-vj9zc\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.857068 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.856936 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlwsr\" (UniqueName: \"kubernetes.io/projected/3f5f78d0-b661-4846-a078-e80aefc88592-kube-api-access-hlwsr\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.857068 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.856951 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb7866b-4998-420e-9afb-29ce3903955c-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:19.857068 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:19.856964 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5f78d0-b661-4846-a078-e80aefc88592-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:20.412248 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.412214 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" Oct 31 04:07:20.412248 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.412237 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt" event={"ID":"dcb7866b-4998-420e-9afb-29ce3903955c","Type":"ContainerDied","Data":"9abc14ffc95bbf1468f88ca7772664412786effd03463dcd6d85961624aaef8a"} Oct 31 04:07:20.412800 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.412286 4426 scope.go:122] "RemoveContainer" containerID="f33cf7346e68bfe4c8345ed244f29f129f07bc13ffa712be663300fd90b75f77" Oct 31 04:07:20.413993 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.413962 4426 generic.go:356] "Generic (PLEG): container finished" podID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerID="bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360" exitCode=0 Oct 31 04:07:20.414239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.414000 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" Oct 31 04:07:20.414239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.414154 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" event={"ID":"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c","Type":"ContainerDied","Data":"bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360"} Oct 31 04:07:20.414239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.414200 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5" event={"ID":"638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c","Type":"ContainerDied","Data":"80d34aab70e9b446c4f5c03820fdc6088b0f59609620c52e37c4ea02b36b1d21"} Oct 31 04:07:20.415894 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.415867 4426 generic.go:356] "Generic (PLEG): container finished" podID="3f5f78d0-b661-4846-a078-e80aefc88592" containerID="7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e" exitCode=0 Oct 31 04:07:20.416025 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.415920 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" Oct 31 04:07:20.416025 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.415956 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" event={"ID":"3f5f78d0-b661-4846-a078-e80aefc88592","Type":"ContainerDied","Data":"7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e"} Oct 31 04:07:20.416181 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.416023 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2" event={"ID":"3f5f78d0-b661-4846-a078-e80aefc88592","Type":"ContainerDied","Data":"1668cb5e77ed1a706415b12716e695ff6d6d77b9670beb037ff61c111c92654a"} Oct 31 04:07:20.429108 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.429090 4426 scope.go:122] "RemoveContainer" containerID="bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360" Oct 31 04:07:20.439395 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.439371 4426 scope.go:122] "RemoveContainer" containerID="bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360" Oct 31 04:07:20.439689 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:07:20.439664 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360\": container with ID starting with bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360 not found: ID does not exist" containerID="bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360" Oct 31 04:07:20.439772 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.439699 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360"} err="failed to get container status \"bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360\": rpc error: code = NotFound desc = could not find container \"bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360\": container with ID starting with bdeed9ab44c03e5061d7aa069a5ed3b8287297397d4b3154d4bac93a1fa57360 not found: ID does not exist" Oct 31 04:07:20.439772 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.439718 4426 scope.go:122] "RemoveContainer" containerID="7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e" Oct 31 04:07:20.449315 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.449292 4426 scope.go:122] "RemoveContainer" containerID="7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e" Oct 31 04:07:20.449621 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:07:20.449598 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e\": container with ID starting with 7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e not found: ID does not exist" containerID="7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e" Oct 31 04:07:20.449692 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.449632 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e"} err="failed to get container status \"7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e\": rpc error: code = NotFound desc = could not find container \"7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e\": container with ID starting with 7a55fb8a60d3b7461091e637790a8ce28feb29bb0ab3b5f61a5989ec5870343e not found: ID does not exist" Oct 31 04:07:20.568307 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.568270 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2"] Oct 31 04:07:20.573699 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.573670 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/premium-w2-a64fd2-kserve-7954d4f997-gwqc2"] Oct 31 04:07:20.718348 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.718251 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt"] Oct 31 04:07:20.722210 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.722185 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/distinct-2-w2-a64fd2-kserve-699c8d999-64dpt"] Oct 31 04:07:20.824708 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.824670 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5"] Oct 31 04:07:20.831059 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.831033 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a64fd2/simulator-w2-a64fd2-kserve-79c95fcf44-kdgv5"] Oct 31 04:07:20.855819 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.855784 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm"] Oct 31 04:07:20.856402 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856387 4426 cpu_manager.go:414] "containerMap: removing container" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" Oct 31 04:07:20.856478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856402 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" Oct 31 04:07:20.856478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856415 4426 cpu_manager.go:414] "containerMap: removing container" podUID="b7614726-c3ed-48eb-9a1b-c0554cf3fe10" containerName="revoke-keys" Oct 31 04:07:20.856478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856422 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7614726-c3ed-48eb-9a1b-c0554cf3fe10" containerName="revoke-keys" Oct 31 04:07:20.856478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856434 4426 cpu_manager.go:414] "containerMap: removing container" podUID="73bb3d25-ee33-4991-88ec-9f84ff8de89b" containerName="revoke-keys" Oct 31 04:07:20.856478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856440 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bb3d25-ee33-4991-88ec-9f84ff8de89b" containerName="revoke-keys" Oct 31 04:07:20.856478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856451 4426 cpu_manager.go:414] "containerMap: removing container" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" Oct 31 04:07:20.856478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856456 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" Oct 31 04:07:20.856478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856463 4426 cpu_manager.go:414] "containerMap: removing container" podUID="91986eb8-661e-4eb2-8c70-077aa1ed61b8" containerName="payload-processing" Oct 31 04:07:20.856478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856468 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91986eb8-661e-4eb2-8c70-077aa1ed61b8" containerName="payload-processing" Oct 31 04:07:20.856478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856483 4426 cpu_manager.go:414] "containerMap: removing container" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856487 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856495 4426 cpu_manager.go:414] "containerMap: removing container" podUID="55f04685-9b38-494b-a38d-560e2411be34" containerName="revoke-keys" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856499 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f04685-9b38-494b-a38d-560e2411be34" containerName="revoke-keys" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856507 4426 cpu_manager.go:414] "containerMap: removing container" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856512 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856522 4426 cpu_manager.go:414] "containerMap: removing container" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerName="istio-proxy" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856527 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerName="istio-proxy" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856604 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bb3d25-ee33-4991-88ec-9f84ff8de89b" containerName="revoke-keys" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856613 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" containerName="main" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856620 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="91986eb8-661e-4eb2-8c70-077aa1ed61b8" containerName="payload-processing" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856628 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" containerName="istio-proxy" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856634 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f04685-9b38-494b-a38d-560e2411be34" containerName="revoke-keys" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856639 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" containerName="main" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856645 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5412c3-19d3-4d81-b59a-20c482bc8b75" containerName="istio-proxy" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856653 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" containerName="main" Oct 31 04:07:20.856981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.856659 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7614726-c3ed-48eb-9a1b-c0554cf3fe10" containerName="revoke-keys" Oct 31 04:07:20.861720 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.861700 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:20.911084 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.911045 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:07:20.911325 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.911094 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:07:20.911409 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.911362 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:07:20.911674 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.911654 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:07:20.915839 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.915815 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm"] Oct 31 04:07:20.972755 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.972654 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p65ph\" (UniqueName: \"kubernetes.io/projected/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-kube-api-access-p65ph\") pod \"maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:20.972926 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.972879 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:20.973001 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:20.972984 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:21.074290 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.074253 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65ph\" (UniqueName: \"kubernetes.io/projected/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-kube-api-access-p65ph\") pod \"maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:21.074504 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.074390 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:21.074504 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.074437 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:21.075057 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.075026 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:21.076816 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.076795 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:21.132354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.132317 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65ph\" (UniqueName: \"kubernetes.io/projected/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-kube-api-access-p65ph\") pod \"maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:21.133364 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.133336 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5f78d0-b661-4846-a078-e80aefc88592" path="/var/lib/kubelet/pods/3f5f78d0-b661-4846-a078-e80aefc88592/volumes" Oct 31 04:07:21.133810 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.133796 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c" path="/var/lib/kubelet/pods/638fac5f-cc53-4bb2-b6c8-6fd2f0ace35c/volumes" Oct 31 04:07:21.134255 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.134239 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5462e40-4af4-4dc9-9a84-4169c3e9fd3b" path="/var/lib/kubelet/pods/d5462e40-4af4-4dc9-9a84-4169c3e9fd3b/volumes" Oct 31 04:07:21.134983 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.134967 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb7866b-4998-420e-9afb-29ce3903955c" path="/var/lib/kubelet/pods/dcb7866b-4998-420e-9afb-29ce3903955c/volumes" Oct 31 04:07:21.175803 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.175768 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:21.555112 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:21.555028 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm"] Oct 31 04:07:21.559538 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:07:21.559513 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790e3ed6_56d7_41cf_a6b4_c0d0aa6608ee.slice/crio-82c2466c1ce023c4e76ce76457bedd783f52cb9388515ec4475f99134fc5328f WatchSource:0}: Error finding container 82c2466c1ce023c4e76ce76457bedd783f52cb9388515ec4475f99134fc5328f: Status 404 returned error can't find the container with id 82c2466c1ce023c4e76ce76457bedd783f52cb9388515ec4475f99134fc5328f Oct 31 04:07:22.446720 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:22.446676 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" event={"ID":"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee","Type":"ContainerStarted","Data":"aaba5cd47481409894f38f0a70c82bffc63f0e12593766b3d26b0578021c1ed5"} Oct 31 04:07:22.446955 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:22.446780 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" event={"ID":"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee","Type":"ContainerStarted","Data":"82c2466c1ce023c4e76ce76457bedd783f52cb9388515ec4475f99134fc5328f"} Oct 31 04:07:22.557882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:22.557831 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" podStartSLOduration=2.5578159620000003 podStartE2EDuration="2.557815962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:07:22.554461738 +0000 UTC m=+2339.555086965" watchObservedRunningTime="2026-10-31 04:07:22.557815962 +0000 UTC m=+2339.558441190" Oct 31 04:07:23.456533 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:23.456495 4426 generic.go:356] "Generic (PLEG): container finished" podID="790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee" containerID="aaba5cd47481409894f38f0a70c82bffc63f0e12593766b3d26b0578021c1ed5" exitCode=0 Oct 31 04:07:23.456738 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:23.456582 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" event={"ID":"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee","Type":"ContainerDied","Data":"aaba5cd47481409894f38f0a70c82bffc63f0e12593766b3d26b0578021c1ed5"} Oct 31 04:07:24.641464 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:24.641433 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:24.713559 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:24.713526 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-kube-api-access-p65ph\" (UniqueName: \"kubernetes.io/projected/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-kube-api-access-p65ph\") pod \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " Oct 31 04:07:24.713740 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:24.713622 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-openshift-ca\") pod \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " Oct 31 04:07:24.713790 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:24.713756 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-xks-ca\") pod \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\" (UID: \"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee\") " Oct 31 04:07:24.714043 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:24.714004 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-openshift-ca" pod "790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee" (UID: "790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:24.714273 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:24.714253 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:24.715840 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:24.715812 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-kube-api-access-p65ph" pod "790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee" (UID: "790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee"). InnerVolumeSpecName "kube-api-access-p65ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:24.716221 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:24.716157 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-xks-ca" pod "790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee" (UID: "790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:24.815248 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:24.815192 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:24.815248 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:24.815238 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p65ph\" (UniqueName: \"kubernetes.io/projected/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee-kube-api-access-p65ph\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:25.475998 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:25.475906 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" Oct 31 04:07:25.475998 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:25.475937 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm" event={"ID":"790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee","Type":"ContainerDied","Data":"82c2466c1ce023c4e76ce76457bedd783f52cb9388515ec4475f99134fc5328f"} Oct 31 04:07:25.475998 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:25.475993 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c2466c1ce023c4e76ce76457bedd783f52cb9388515ec4475f99134fc5328f" Oct 31 04:07:27.326568 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:27.326523 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" podUID="89cd7dd5-9af7-40f3-b01f-96eb4f64aae6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:07:32.847179 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:07:32.847108 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:07:33.705309 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.705266 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq"] Oct 31 04:07:33.705908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.705891 4426 cpu_manager.go:414] "containerMap: removing container" podUID="790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee" containerName="revoke-keys" Oct 31 04:07:33.705908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.705909 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee" containerName="revoke-keys" Oct 31 04:07:33.706047 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.706001 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee" containerName="revoke-keys" Oct 31 04:07:33.711481 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.711461 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:33.742704 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.742672 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rgsf\" (UniqueName: \"kubernetes.io/projected/c46a697e-7668-4ea6-9e57-a726b4b8edd6-kube-api-access-9rgsf\") pod \"maas-api-revoke-sub-5d3b696c-cxsvq\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:33.742842 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.742737 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d3b696c-cxsvq\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:33.742842 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.742805 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d3b696c-cxsvq\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:33.761009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.760977 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl"] Oct 31 04:07:33.766183 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.766153 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:33.766628 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.766602 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq"] Oct 31 04:07:33.803276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.803240 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:07:33.803528 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.803254 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:07:33.803528 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.803299 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:07:33.807387 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.807358 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:07:33.822064 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.822033 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl"] Oct 31 04:07:33.843869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.843823 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rgsf\" (UniqueName: \"kubernetes.io/projected/c46a697e-7668-4ea6-9e57-a726b4b8edd6-kube-api-access-9rgsf\") pod \"maas-api-revoke-sub-5d3b696c-cxsvq\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:33.844047 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.843891 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849dc88c-xbxsl\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:33.844047 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.843958 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d3b696c-cxsvq\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:33.844047 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.843989 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d3b696c-cxsvq\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:33.844396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.844372 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nltzv\" (UniqueName: \"kubernetes.io/projected/aba6ac38-84db-419c-899f-af8ae6904dc1-kube-api-access-nltzv\") pod \"maas-api-revoke-sub-849dc88c-xbxsl\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:33.844535 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.844510 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849dc88c-xbxsl\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:33.844915 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.844872 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d3b696c-cxsvq\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:33.846654 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.846629 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d3b696c-cxsvq\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:33.901695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.901656 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rgsf\" (UniqueName: \"kubernetes.io/projected/c46a697e-7668-4ea6-9e57-a726b4b8edd6-kube-api-access-9rgsf\") pod \"maas-api-revoke-sub-5d3b696c-cxsvq\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:33.949155 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.945812 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849dc88c-xbxsl\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:33.949155 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.945910 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nltzv\" (UniqueName: \"kubernetes.io/projected/aba6ac38-84db-419c-899f-af8ae6904dc1-kube-api-access-nltzv\") pod \"maas-api-revoke-sub-849dc88c-xbxsl\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:33.949155 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.945970 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849dc88c-xbxsl\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:33.950377 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.950340 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849dc88c-xbxsl\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:33.950788 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:33.950754 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849dc88c-xbxsl\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:34.008381 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:34.008303 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nltzv\" (UniqueName: \"kubernetes.io/projected/aba6ac38-84db-419c-899f-af8ae6904dc1-kube-api-access-nltzv\") pod \"maas-api-revoke-sub-849dc88c-xbxsl\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:34.026444 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:34.026412 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:34.081386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:34.081345 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:34.411924 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:34.411862 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq"] Oct 31 04:07:34.416613 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:07:34.416563 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46a697e_7668_4ea6_9e57_a726b4b8edd6.slice/crio-53773284ee9e1b3b37212f84646a315c9a49bc1c6e6023b8e342f670f000ebf1 WatchSource:0}: Error finding container 53773284ee9e1b3b37212f84646a315c9a49bc1c6e6023b8e342f670f000ebf1: Status 404 returned error can't find the container with id 53773284ee9e1b3b37212f84646a315c9a49bc1c6e6023b8e342f670f000ebf1 Oct 31 04:07:34.529367 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:34.529335 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl"] Oct 31 04:07:34.533094 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:07:34.533062 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba6ac38_84db_419c_899f_af8ae6904dc1.slice/crio-66a48a2823413859f6010918b8a9cbfd29c8b364eb307c73a4a598ff44b67f80 WatchSource:0}: Error finding container 66a48a2823413859f6010918b8a9cbfd29c8b364eb307c73a4a598ff44b67f80: Status 404 returned error can't find the container with id 66a48a2823413859f6010918b8a9cbfd29c8b364eb307c73a4a598ff44b67f80 Oct 31 04:07:34.546766 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:34.546734 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" event={"ID":"aba6ac38-84db-419c-899f-af8ae6904dc1","Type":"ContainerStarted","Data":"66a48a2823413859f6010918b8a9cbfd29c8b364eb307c73a4a598ff44b67f80"} Oct 31 04:07:34.548269 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:34.548241 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" event={"ID":"c46a697e-7668-4ea6-9e57-a726b4b8edd6","Type":"ContainerStarted","Data":"53773284ee9e1b3b37212f84646a315c9a49bc1c6e6023b8e342f670f000ebf1"} Oct 31 04:07:35.557457 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:35.557363 4426 generic.go:356] "Generic (PLEG): container finished" podID="aba6ac38-84db-419c-899f-af8ae6904dc1" containerID="fce7a42e787f1bb09b70eb6f2271e65f3e37df6f60c94adfb7f576790a0a69e5" exitCode=0 Oct 31 04:07:35.557457 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:35.557441 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" event={"ID":"aba6ac38-84db-419c-899f-af8ae6904dc1","Type":"ContainerDied","Data":"fce7a42e787f1bb09b70eb6f2271e65f3e37df6f60c94adfb7f576790a0a69e5"} Oct 31 04:07:35.558987 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:35.558959 4426 generic.go:356] "Generic (PLEG): container finished" podID="c46a697e-7668-4ea6-9e57-a726b4b8edd6" containerID="a91162c79ecad5b28341396ae918c16d740d330ad1fdc76c25048164c71aaacc" exitCode=0 Oct 31 04:07:35.559097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:35.559012 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" event={"ID":"c46a697e-7668-4ea6-9e57-a726b4b8edd6","Type":"ContainerDied","Data":"a91162c79ecad5b28341396ae918c16d740d330ad1fdc76c25048164c71aaacc"} Oct 31 04:07:36.580914 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.580876 4426 generic.go:356] "Generic (PLEG): container finished" podID="89cd7dd5-9af7-40f3-b01f-96eb4f64aae6" containerID="c809e023f856a11cecbb5885eaa9015a97e117d06e93bd20fa26e94ad36541c8" exitCode=0 Oct 31 04:07:36.581379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.581312 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" event={"ID":"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6","Type":"ContainerDied","Data":"c809e023f856a11cecbb5885eaa9015a97e117d06e93bd20fa26e94ad36541c8"} Oct 31 04:07:36.683171 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.682786 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 04:07:36.761869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.761841 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:36.769485 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.769459 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:36.780631 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.780604 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-kube-api-access-66mbg\" (UniqueName: \"kubernetes.io/projected/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-kube-api-access-66mbg\") pod \"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6\" (UID: \"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6\") " Oct 31 04:07:36.780774 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.780717 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-plugins-config-volume\") pod \"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6\" (UID: \"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6\") " Oct 31 04:07:36.781088 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.781061 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-plugins-config-volume" pod "89cd7dd5-9af7-40f3-b01f-96eb4f64aae6" (UID: "89cd7dd5-9af7-40f3-b01f-96eb4f64aae6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:36.782660 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.782640 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-kube-api-access-66mbg" pod "89cd7dd5-9af7-40f3-b01f-96eb4f64aae6" (UID: "89cd7dd5-9af7-40f3-b01f-96eb4f64aae6"). InnerVolumeSpecName "kube-api-access-66mbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:36.881612 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.881573 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-openshift-ca\") pod \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " Oct 31 04:07:36.881825 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.881700 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-xks-ca\") pod \"aba6ac38-84db-419c-899f-af8ae6904dc1\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " Oct 31 04:07:36.881825 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.881750 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-xks-ca\") pod \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " Oct 31 04:07:36.881825 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.881793 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-openshift-ca\") pod \"aba6ac38-84db-419c-899f-af8ae6904dc1\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " Oct 31 04:07:36.881988 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.881828 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba6ac38-84db-419c-899f-af8ae6904dc1-kube-api-access-nltzv\" (UniqueName: \"kubernetes.io/projected/aba6ac38-84db-419c-899f-af8ae6904dc1-kube-api-access-nltzv\") pod \"aba6ac38-84db-419c-899f-af8ae6904dc1\" (UID: \"aba6ac38-84db-419c-899f-af8ae6904dc1\") " Oct 31 04:07:36.881988 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.881882 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46a697e-7668-4ea6-9e57-a726b4b8edd6-kube-api-access-9rgsf\" (UniqueName: \"kubernetes.io/projected/c46a697e-7668-4ea6-9e57-a726b4b8edd6-kube-api-access-9rgsf\") pod \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\" (UID: \"c46a697e-7668-4ea6-9e57-a726b4b8edd6\") " Oct 31 04:07:36.881988 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.881968 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-openshift-ca" pod "c46a697e-7668-4ea6-9e57-a726b4b8edd6" (UID: "c46a697e-7668-4ea6-9e57-a726b4b8edd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:36.882282 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.882254 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-openshift-ca" pod "aba6ac38-84db-419c-899f-af8ae6904dc1" (UID: "aba6ac38-84db-419c-899f-af8ae6904dc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:36.882356 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.882335 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66mbg\" (UniqueName: \"kubernetes.io/projected/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-kube-api-access-66mbg\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:36.882408 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.882366 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:36.882408 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.882385 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:36.882408 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.882401 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:36.883964 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.883935 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-xks-ca" pod "aba6ac38-84db-419c-899f-af8ae6904dc1" (UID: "aba6ac38-84db-419c-899f-af8ae6904dc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:36.884120 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.884077 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46a697e-7668-4ea6-9e57-a726b4b8edd6-kube-api-access-9rgsf" pod "c46a697e-7668-4ea6-9e57-a726b4b8edd6" (UID: "c46a697e-7668-4ea6-9e57-a726b4b8edd6"). InnerVolumeSpecName "kube-api-access-9rgsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:36.884326 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.884304 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba6ac38-84db-419c-899f-af8ae6904dc1-kube-api-access-nltzv" pod "aba6ac38-84db-419c-899f-af8ae6904dc1" (UID: "aba6ac38-84db-419c-899f-af8ae6904dc1"). InnerVolumeSpecName "kube-api-access-nltzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:36.884395 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.884335 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-xks-ca" pod "c46a697e-7668-4ea6-9e57-a726b4b8edd6" (UID: "c46a697e-7668-4ea6-9e57-a726b4b8edd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:36.983817 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.983721 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba6ac38-84db-419c-899f-af8ae6904dc1-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:36.983817 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.983753 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46a697e-7668-4ea6-9e57-a726b4b8edd6-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:36.983817 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.983776 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nltzv\" (UniqueName: \"kubernetes.io/projected/aba6ac38-84db-419c-899f-af8ae6904dc1-kube-api-access-nltzv\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:36.983817 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:36.983790 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rgsf\" (UniqueName: \"kubernetes.io/projected/c46a697e-7668-4ea6-9e57-a726b4b8edd6-kube-api-access-9rgsf\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:37.590827 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:37.590793 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" Oct 31 04:07:37.591318 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:37.590791 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl" event={"ID":"aba6ac38-84db-419c-899f-af8ae6904dc1","Type":"ContainerDied","Data":"66a48a2823413859f6010918b8a9cbfd29c8b364eb307c73a4a598ff44b67f80"} Oct 31 04:07:37.591318 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:37.590935 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a48a2823413859f6010918b8a9cbfd29c8b364eb307c73a4a598ff44b67f80" Oct 31 04:07:37.592316 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:37.592296 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" Oct 31 04:07:37.592451 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:37.592318 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq" event={"ID":"c46a697e-7668-4ea6-9e57-a726b4b8edd6","Type":"ContainerDied","Data":"53773284ee9e1b3b37212f84646a315c9a49bc1c6e6023b8e342f670f000ebf1"} Oct 31 04:07:37.592451 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:37.592353 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53773284ee9e1b3b37212f84646a315c9a49bc1c6e6023b8e342f670f000ebf1" Oct 31 04:07:37.593981 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:37.593965 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" Oct 31 04:07:37.594159 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:37.594003 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq" event={"ID":"89cd7dd5-9af7-40f3-b01f-96eb4f64aae6","Type":"ContainerDied","Data":"b541b1887b864ffe9508eb798d56b20a6a37593e85e10d3f8d33eb5e2c747472"} Oct 31 04:07:37.594159 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:37.594055 4426 scope.go:122] "RemoveContainer" containerID="c809e023f856a11cecbb5885eaa9015a97e117d06e93bd20fa26e94ad36541c8" Oct 31 04:07:37.822028 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:37.821936 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq"] Oct 31 04:07:37.867596 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:37.867562 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a64fd2-79f5f9f94f-qsvpq"] Oct 31 04:07:39.148592 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:39.148560 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cd7dd5-9af7-40f3-b01f-96eb4f64aae6" path="/var/lib/kubelet/pods/89cd7dd5-9af7-40f3-b01f-96eb4f64aae6/volumes" Oct 31 04:07:54.234190 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.234142 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8"] Oct 31 04:07:54.234882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.234755 4426 cpu_manager.go:414] "containerMap: removing container" podUID="aba6ac38-84db-419c-899f-af8ae6904dc1" containerName="revoke-keys" Oct 31 04:07:54.234882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.234768 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba6ac38-84db-419c-899f-af8ae6904dc1" containerName="revoke-keys" Oct 31 04:07:54.234882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.234783 4426 cpu_manager.go:414] "containerMap: removing container" podUID="89cd7dd5-9af7-40f3-b01f-96eb4f64aae6" containerName="payload-processing" Oct 31 04:07:54.234882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.234789 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cd7dd5-9af7-40f3-b01f-96eb4f64aae6" containerName="payload-processing" Oct 31 04:07:54.234882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.234803 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c46a697e-7668-4ea6-9e57-a726b4b8edd6" containerName="revoke-keys" Oct 31 04:07:54.234882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.234808 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46a697e-7668-4ea6-9e57-a726b4b8edd6" containerName="revoke-keys" Oct 31 04:07:54.235236 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.234897 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba6ac38-84db-419c-899f-af8ae6904dc1" containerName="revoke-keys" Oct 31 04:07:54.235236 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.234907 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cd7dd5-9af7-40f3-b01f-96eb4f64aae6" containerName="payload-processing" Oct 31 04:07:54.235236 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.234914 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46a697e-7668-4ea6-9e57-a726b4b8edd6" containerName="revoke-keys" Oct 31 04:07:54.238720 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.238693 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:54.289015 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.288983 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p"] Oct 31 04:07:54.293847 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.293821 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:54.294802 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.294780 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8"] Oct 31 04:07:54.330473 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.330435 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:07:54.330733 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.330551 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:07:54.330733 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.330449 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:07:54.334161 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.334120 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:07:54.348090 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.348059 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p"] Oct 31 04:07:54.399203 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.399159 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd58ce20-k2js8\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:54.399386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.399284 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61276fbd-l877p\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:54.399386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.399323 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd58ce20-k2js8\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:54.399386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.399343 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rk7\" (UniqueName: \"kubernetes.io/projected/ab6c7b8b-f227-415c-9149-28650db67b48-kube-api-access-g6rk7\") pod \"maas-api-revoke-sub-61276fbd-l877p\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:54.399386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.399360 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61276fbd-l877p\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:54.399557 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.399543 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjvck\" (UniqueName: \"kubernetes.io/projected/e9c4f03e-0e39-4532-8ef1-3904dd026460-kube-api-access-hjvck\") pod \"maas-api-revoke-sub-cd58ce20-k2js8\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:54.503014 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.502920 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd58ce20-k2js8\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:54.503014 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.502978 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61276fbd-l877p\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:54.503014 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.503012 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd58ce20-k2js8\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:54.503387 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.503030 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rk7\" (UniqueName: \"kubernetes.io/projected/ab6c7b8b-f227-415c-9149-28650db67b48-kube-api-access-g6rk7\") pod \"maas-api-revoke-sub-61276fbd-l877p\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:54.503387 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.503046 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61276fbd-l877p\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:54.503387 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.503136 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjvck\" (UniqueName: \"kubernetes.io/projected/e9c4f03e-0e39-4532-8ef1-3904dd026460-kube-api-access-hjvck\") pod \"maas-api-revoke-sub-cd58ce20-k2js8\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:54.503872 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.503848 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd58ce20-k2js8\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:54.504048 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.504015 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61276fbd-l877p\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:54.505682 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.505653 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd58ce20-k2js8\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:54.505820 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.505762 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61276fbd-l877p\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:54.562616 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.562583 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rk7\" (UniqueName: \"kubernetes.io/projected/ab6c7b8b-f227-415c-9149-28650db67b48-kube-api-access-g6rk7\") pod \"maas-api-revoke-sub-61276fbd-l877p\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:54.562616 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.562605 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjvck\" (UniqueName: \"kubernetes.io/projected/e9c4f03e-0e39-4532-8ef1-3904dd026460-kube-api-access-hjvck\") pod \"maas-api-revoke-sub-cd58ce20-k2js8\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:54.612740 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.612704 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:54.853204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:54.853105 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:55.004632 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:55.004597 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p"] Oct 31 04:07:55.008752 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:07:55.008722 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6c7b8b_f227_415c_9149_28650db67b48.slice/crio-bf7c22c48b2df3a729de05356b8a3f40189314ed35c3d5b720527ab5d6a7b513 WatchSource:0}: Error finding container bf7c22c48b2df3a729de05356b8a3f40189314ed35c3d5b720527ab5d6a7b513: Status 404 returned error can't find the container with id bf7c22c48b2df3a729de05356b8a3f40189314ed35c3d5b720527ab5d6a7b513 Oct 31 04:07:55.189730 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:55.189697 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4"] Oct 31 04:07:55.194610 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:55.194574 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d7b7bc-2l6x4"] Oct 31 04:07:55.261555 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:55.261524 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8"] Oct 31 04:07:55.263535 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:07:55.263500 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c4f03e_0e39_4532_8ef1_3904dd026460.slice/crio-39f3a14045d030453f35053b1aab71cc5a5d76bb386baeeaaa016225c8f5647c WatchSource:0}: Error finding container 39f3a14045d030453f35053b1aab71cc5a5d76bb386baeeaaa016225c8f5647c: Status 404 returned error can't find the container with id 39f3a14045d030453f35053b1aab71cc5a5d76bb386baeeaaa016225c8f5647c Oct 31 04:07:55.751864 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:55.751814 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" event={"ID":"ab6c7b8b-f227-415c-9149-28650db67b48","Type":"ContainerStarted","Data":"c92f2ae5c57464d27655e885e0fa9222f7e9df69b9827ab5c0f7fb194f530c0f"} Oct 31 04:07:55.752088 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:55.751933 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" event={"ID":"ab6c7b8b-f227-415c-9149-28650db67b48","Type":"ContainerStarted","Data":"bf7c22c48b2df3a729de05356b8a3f40189314ed35c3d5b720527ab5d6a7b513"} Oct 31 04:07:55.753774 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:55.753745 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" event={"ID":"e9c4f03e-0e39-4532-8ef1-3904dd026460","Type":"ContainerStarted","Data":"3a35f3206519312e580ed5680e99caa314478e095891b49fdda6b7f16bed5530"} Oct 31 04:07:55.753885 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:55.753799 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" event={"ID":"e9c4f03e-0e39-4532-8ef1-3904dd026460","Type":"ContainerStarted","Data":"39f3a14045d030453f35053b1aab71cc5a5d76bb386baeeaaa016225c8f5647c"} Oct 31 04:07:55.864464 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:55.864355 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" podStartSLOduration=1.8643403950000002 podStartE2EDuration="1.864340395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:07:55.859978663 +0000 UTC m=+2372.860603891" watchObservedRunningTime="2026-10-31 04:07:55.864340395 +0000 UTC m=+2372.864965622" Oct 31 04:07:55.977938 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:55.977746 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" podStartSLOduration=1.977732757 podStartE2EDuration="1.977732757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:07:55.976309602 +0000 UTC m=+2372.976934832" watchObservedRunningTime="2026-10-31 04:07:55.977732757 +0000 UTC m=+2372.978357986" Oct 31 04:07:56.763881 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:56.763840 4426 generic.go:356] "Generic (PLEG): container finished" podID="e9c4f03e-0e39-4532-8ef1-3904dd026460" containerID="3a35f3206519312e580ed5680e99caa314478e095891b49fdda6b7f16bed5530" exitCode=0 Oct 31 04:07:56.764335 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:56.763912 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" event={"ID":"e9c4f03e-0e39-4532-8ef1-3904dd026460","Type":"ContainerDied","Data":"3a35f3206519312e580ed5680e99caa314478e095891b49fdda6b7f16bed5530"} Oct 31 04:07:56.765973 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:56.765941 4426 generic.go:356] "Generic (PLEG): container finished" podID="ab6c7b8b-f227-415c-9149-28650db67b48" containerID="c92f2ae5c57464d27655e885e0fa9222f7e9df69b9827ab5c0f7fb194f530c0f" exitCode=0 Oct 31 04:07:56.766100 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:56.766023 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" event={"ID":"ab6c7b8b-f227-415c-9149-28650db67b48","Type":"ContainerDied","Data":"c92f2ae5c57464d27655e885e0fa9222f7e9df69b9827ab5c0f7fb194f530c0f"} Oct 31 04:07:57.129650 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:57.129571 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d61453-63b4-4275-97f2-8fd85055f80a" path="/var/lib/kubelet/pods/16d61453-63b4-4275-97f2-8fd85055f80a/volumes" Oct 31 04:07:57.969004 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:57.968973 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:57.971442 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:57.971422 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:58.044526 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.044428 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-openshift-ca\") pod \"ab6c7b8b-f227-415c-9149-28650db67b48\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " Oct 31 04:07:58.044710 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.044568 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c4f03e-0e39-4532-8ef1-3904dd026460-kube-api-access-hjvck\" (UniqueName: \"kubernetes.io/projected/e9c4f03e-0e39-4532-8ef1-3904dd026460-kube-api-access-hjvck\") pod \"e9c4f03e-0e39-4532-8ef1-3904dd026460\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " Oct 31 04:07:58.044710 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.044612 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab6c7b8b-f227-415c-9149-28650db67b48-kube-api-access-g6rk7\" (UniqueName: \"kubernetes.io/projected/ab6c7b8b-f227-415c-9149-28650db67b48-kube-api-access-g6rk7\") pod \"ab6c7b8b-f227-415c-9149-28650db67b48\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " Oct 31 04:07:58.044842 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.044705 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-xks-ca\") pod \"ab6c7b8b-f227-415c-9149-28650db67b48\" (UID: \"ab6c7b8b-f227-415c-9149-28650db67b48\") " Oct 31 04:07:58.044842 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.044745 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-openshift-ca\") pod \"e9c4f03e-0e39-4532-8ef1-3904dd026460\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " Oct 31 04:07:58.044842 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.044780 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-xks-ca\") pod \"e9c4f03e-0e39-4532-8ef1-3904dd026460\" (UID: \"e9c4f03e-0e39-4532-8ef1-3904dd026460\") " Oct 31 04:07:58.045004 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.044971 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-openshift-ca" pod "ab6c7b8b-f227-415c-9149-28650db67b48" (UID: "ab6c7b8b-f227-415c-9149-28650db67b48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:58.045378 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.045314 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:58.045626 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.045601 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-openshift-ca" pod "e9c4f03e-0e39-4532-8ef1-3904dd026460" (UID: "e9c4f03e-0e39-4532-8ef1-3904dd026460"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:58.047487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.047426 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-xks-ca" pod "ab6c7b8b-f227-415c-9149-28650db67b48" (UID: "ab6c7b8b-f227-415c-9149-28650db67b48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:58.047487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.047461 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-xks-ca" pod "e9c4f03e-0e39-4532-8ef1-3904dd026460" (UID: "e9c4f03e-0e39-4532-8ef1-3904dd026460"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:58.047882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.047851 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6c7b8b-f227-415c-9149-28650db67b48-kube-api-access-g6rk7" pod "ab6c7b8b-f227-415c-9149-28650db67b48" (UID: "ab6c7b8b-f227-415c-9149-28650db67b48"). InnerVolumeSpecName "kube-api-access-g6rk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:58.047973 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.047913 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c4f03e-0e39-4532-8ef1-3904dd026460-kube-api-access-hjvck" pod "e9c4f03e-0e39-4532-8ef1-3904dd026460" (UID: "e9c4f03e-0e39-4532-8ef1-3904dd026460"). InnerVolumeSpecName "kube-api-access-hjvck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:58.146814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.146566 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjvck\" (UniqueName: \"kubernetes.io/projected/e9c4f03e-0e39-4532-8ef1-3904dd026460-kube-api-access-hjvck\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:58.146814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.146602 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6rk7\" (UniqueName: \"kubernetes.io/projected/ab6c7b8b-f227-415c-9149-28650db67b48-kube-api-access-g6rk7\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:58.146814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.146617 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab6c7b8b-f227-415c-9149-28650db67b48-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:58.146814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.146631 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:58.146814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.146646 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c4f03e-0e39-4532-8ef1-3904dd026460-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:58.783230 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.783198 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" Oct 31 04:07:58.783230 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.783219 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p" event={"ID":"ab6c7b8b-f227-415c-9149-28650db67b48","Type":"ContainerDied","Data":"bf7c22c48b2df3a729de05356b8a3f40189314ed35c3d5b720527ab5d6a7b513"} Oct 31 04:07:58.783499 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.783285 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf7c22c48b2df3a729de05356b8a3f40189314ed35c3d5b720527ab5d6a7b513" Oct 31 04:07:58.784883 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.784859 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" Oct 31 04:07:58.785018 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.784886 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8" event={"ID":"e9c4f03e-0e39-4532-8ef1-3904dd026460","Type":"ContainerDied","Data":"39f3a14045d030453f35053b1aab71cc5a5d76bb386baeeaaa016225c8f5647c"} Oct 31 04:07:58.785018 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:07:58.784930 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39f3a14045d030453f35053b1aab71cc5a5d76bb386baeeaaa016225c8f5647c" Oct 31 04:08:02.849801 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:08:02.849757 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:08:03.066248 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:03.066209 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc"] Oct 31 04:08:03.066500 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:03.066470 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" podUID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" containerName="istio-proxy" containerID="cri-o://17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8" gracePeriod=30 Oct 31 04:08:05.558961 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:05.558914 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:08:08.332290 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.332253 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 04:08:08.465084 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465045 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-data\") pod \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " Oct 31 04:08:08.465311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465136 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-kube-api-access-ljbn8\" (UniqueName: \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-kube-api-access-ljbn8\") pod \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " Oct 31 04:08:08.465311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465231 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-socket\") pod \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " Oct 31 04:08:08.465311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465294 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-certs\") pod \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " Oct 31 04:08:08.465510 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465343 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-credential-socket\") pod \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " Oct 31 04:08:08.465510 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465388 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-podinfo\") pod \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " Oct 31 04:08:08.465510 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465386 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-data" pod "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" (UID: "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:08.465510 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465446 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-token\" (UniqueName: \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-token\") pod \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " Oct 31 04:08:08.465510 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465489 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-envoy\") pod \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " Oct 31 04:08:08.465789 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465578 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istiod-ca-cert\") pod \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\" (UID: \"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa\") " Oct 31 04:08:08.465789 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465630 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-certs" pod "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" (UID: "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:08.465789 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465650 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-credential-socket" pod "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" (UID: "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:08.465975 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.465942 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-socket" pod "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" (UID: "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:08.466160 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.466136 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:08.466257 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.466168 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:08.466257 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.466188 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:08.466257 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.466203 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:08.466420 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.466281 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istiod-ca-cert" pod "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" (UID: "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:08:08.467713 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.467680 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-kube-api-access-ljbn8" pod "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" (UID: "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa"). InnerVolumeSpecName "kube-api-access-ljbn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:08.467876 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.467802 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-token" pod "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" (UID: "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:08.468232 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.468209 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-envoy" pod "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" (UID: "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:08.468547 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.468527 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-podinfo" pod "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" (UID: "aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:08:08.567726 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.567675 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljbn8\" (UniqueName: \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-kube-api-access-ljbn8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:08.567726 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.567721 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:08.567726 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.567734 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:08.567980 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.567745 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:08.567980 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.567755 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:08.885319 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.885210 4426 generic.go:356] "Generic (PLEG): container finished" podID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" containerID="17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8" exitCode=0 Oct 31 04:08:08.885319 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.885301 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" event={"ID":"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa","Type":"ContainerDied","Data":"17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8"} Oct 31 04:08:08.885559 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.885321 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" Oct 31 04:08:08.885559 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.885353 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc" event={"ID":"aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa","Type":"ContainerDied","Data":"3d1ea5690110b30d8151e91318845d9afd3fdeb5d20f587f521b9cf609d7eba2"} Oct 31 04:08:08.885559 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.885375 4426 scope.go:122] "RemoveContainer" containerID="17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8" Oct 31 04:08:08.900295 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.900271 4426 scope.go:122] "RemoveContainer" containerID="17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8" Oct 31 04:08:08.900655 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:08:08.900631 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8\": container with ID starting with 17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8 not found: ID does not exist" containerID="17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8" Oct 31 04:08:08.900739 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:08.900665 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8"} err="failed to get container status \"17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8\": rpc error: code = NotFound desc = could not find container \"17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8\": container with ID starting with 17e88a80a987d7ba00f731d59adbee9efba3beafeaa36c1d3f15f24d637fa0a8 not found: ID does not exist" Oct 31 04:08:09.042546 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:09.042510 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc"] Oct 31 04:08:09.046534 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:09.046507 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d7b7bc-openshift-default-5b48c74cfc-2pwfc"] Oct 31 04:08:09.134523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:09.134477 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" path="/var/lib/kubelet/pods/aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa/volumes" Oct 31 04:08:21.050836 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:21.050787 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g"] Oct 31 04:08:21.095849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:21.095812 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03173b5c-n6z4g"] Oct 31 04:08:21.129527 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:21.129486 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40dbf523-aa55-42e0-a538-a0a5d3ae068a" path="/var/lib/kubelet/pods/40dbf523-aa55-42e0-a538-a0a5d3ae068a/volumes" Oct 31 04:08:23.560993 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:23.560952 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk"] Oct 31 04:08:23.603569 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:23.603537 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-537e34-pmbkk"] Oct 31 04:08:23.627283 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:23.627259 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:08:23.642429 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:23.642400 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:08:24.694873 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:24.694835 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm"] Oct 31 04:08:24.741045 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:24.741003 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada9ed-c7grm"] Oct 31 04:08:25.130187 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:25.130094 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b5cd88-570f-4aa6-995b-a9be1ab07b69" path="/var/lib/kubelet/pods/64b5cd88-570f-4aa6-995b-a9be1ab07b69/volumes" Oct 31 04:08:25.130762 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:25.130742 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee" path="/var/lib/kubelet/pods/790e3ed6-56d7-41cf-a6b4-c0d0aa6608ee/volumes" Oct 31 04:08:30.411606 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:30.411569 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x"] Oct 31 04:08:30.412188 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:30.411944 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" containerID="cri-o://3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef" gracePeriod=30 Oct 31 04:08:32.849173 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:08:32.849120 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:08:32.892742 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.892705 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24"] Oct 31 04:08:32.893403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.893385 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ab6c7b8b-f227-415c-9149-28650db67b48" containerName="revoke-keys" Oct 31 04:08:32.893403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.893403 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab6c7b8b-f227-415c-9149-28650db67b48" containerName="revoke-keys" Oct 31 04:08:32.893561 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.893425 4426 cpu_manager.go:414] "containerMap: removing container" podUID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" containerName="istio-proxy" Oct 31 04:08:32.893561 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.893430 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" containerName="istio-proxy" Oct 31 04:08:32.893561 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.893444 4426 cpu_manager.go:414] "containerMap: removing container" podUID="e9c4f03e-0e39-4532-8ef1-3904dd026460" containerName="revoke-keys" Oct 31 04:08:32.893561 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.893448 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c4f03e-0e39-4532-8ef1-3904dd026460" containerName="revoke-keys" Oct 31 04:08:32.893561 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.893545 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9ad3b0-b2d6-4c1b-b097-d15072fd93aa" containerName="istio-proxy" Oct 31 04:08:32.893561 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.893554 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c4f03e-0e39-4532-8ef1-3904dd026460" containerName="revoke-keys" Oct 31 04:08:32.893747 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.893565 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab6c7b8b-f227-415c-9149-28650db67b48" containerName="revoke-keys" Oct 31 04:08:32.898768 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.898750 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:32.954222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.954184 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24"] Oct 31 04:08:32.962580 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.962526 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3473a4-qwc24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:32.962777 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.962714 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgtvk\" (UniqueName: \"kubernetes.io/projected/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-kube-api-access-bgtvk\") pod \"maas-api-revoke-keys-e2e-api-b-3473a4-qwc24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:32.962863 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.962782 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3473a4-qwc24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:32.994033 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.993992 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:08:32.994033 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.994015 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:08:32.994300 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.994059 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:08:32.994372 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:32.994015 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:08:33.063819 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:33.063773 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgtvk\" (UniqueName: \"kubernetes.io/projected/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-kube-api-access-bgtvk\") pod \"maas-api-revoke-keys-e2e-api-b-3473a4-qwc24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:33.063819 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:33.063823 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3473a4-qwc24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:33.064100 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:33.063906 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3473a4-qwc24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:33.064695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:33.064661 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3473a4-qwc24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:33.066726 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:33.066703 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3473a4-qwc24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:33.121976 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:33.121889 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgtvk\" (UniqueName: \"kubernetes.io/projected/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-kube-api-access-bgtvk\") pod \"maas-api-revoke-keys-e2e-api-b-3473a4-qwc24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:33.212924 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:33.212894 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:33.602001 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:33.601962 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24"] Oct 31 04:08:33.606685 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:08:33.606658 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c5a965_165b_4ed6_8f4e_8d740fe1bd24.slice/crio-14963ad101913e09b1f9420c1aeeb97612bb3fd92c9296e9df9e98b8a99741d1 WatchSource:0}: Error finding container 14963ad101913e09b1f9420c1aeeb97612bb3fd92c9296e9df9e98b8a99741d1: Status 404 returned error can't find the container with id 14963ad101913e09b1f9420c1aeeb97612bb3fd92c9296e9df9e98b8a99741d1 Oct 31 04:08:34.102622 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:34.102579 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" event={"ID":"41c5a965-165b-4ed6-8f4e-8d740fe1bd24","Type":"ContainerStarted","Data":"338ab1861b6a2160ee9319312699d311102fc6ce76b92356aa7a9bbad4e61624"} Oct 31 04:08:34.103169 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:34.102664 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" event={"ID":"41c5a965-165b-4ed6-8f4e-8d740fe1bd24","Type":"ContainerStarted","Data":"14963ad101913e09b1f9420c1aeeb97612bb3fd92c9296e9df9e98b8a99741d1"} Oct 31 04:08:34.216196 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:34.216098 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" podStartSLOduration=2.216075797 podStartE2EDuration="2.216075797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:34.210413553 +0000 UTC m=+2411.211038781" watchObservedRunningTime="2026-10-31 04:08:34.216075797 +0000 UTC m=+2411.216701039" Oct 31 04:08:35.111286 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.111247 4426 generic.go:356] "Generic (PLEG): container finished" podID="41c5a965-165b-4ed6-8f4e-8d740fe1bd24" containerID="338ab1861b6a2160ee9319312699d311102fc6ce76b92356aa7a9bbad4e61624" exitCode=0 Oct 31 04:08:35.111725 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.111332 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" event={"ID":"41c5a965-165b-4ed6-8f4e-8d740fe1bd24","Type":"ContainerDied","Data":"338ab1861b6a2160ee9319312699d311102fc6ce76b92356aa7a9bbad4e61624"} Oct 31 04:08:35.681454 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.681410 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 04:08:35.794859 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.794762 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-credential-socket\") pod \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " Oct 31 04:08:35.794859 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.794851 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-data\") pod \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " Oct 31 04:08:35.795103 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.794899 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-socket\") pod \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " Oct 31 04:08:35.795103 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.794940 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-kube-api-access-clnkb\" (UniqueName: \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-kube-api-access-clnkb\") pod \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " Oct 31 04:08:35.795103 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.795018 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-token\" (UniqueName: \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-token\") pod \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " Oct 31 04:08:35.795103 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.795057 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-envoy\") pod \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " Oct 31 04:08:35.795103 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.795097 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-podinfo\") pod \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " Oct 31 04:08:35.795416 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.795099 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-credential-socket" pod "f8f0fec9-03b7-4707-8d26-f40f5d1d312c" (UID: "f8f0fec9-03b7-4707-8d26-f40f5d1d312c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:35.795416 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.795244 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-certs\") pod \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " Oct 31 04:08:35.795416 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.795256 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-data" pod "f8f0fec9-03b7-4707-8d26-f40f5d1d312c" (UID: "f8f0fec9-03b7-4707-8d26-f40f5d1d312c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:35.795416 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.795268 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-socket" pod "f8f0fec9-03b7-4707-8d26-f40f5d1d312c" (UID: "f8f0fec9-03b7-4707-8d26-f40f5d1d312c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:35.795416 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.795308 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istiod-ca-cert\") pod \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\" (UID: \"f8f0fec9-03b7-4707-8d26-f40f5d1d312c\") " Oct 31 04:08:35.795751 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.795725 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:35.795814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.795756 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:35.795814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.795770 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:35.796107 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.796084 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istiod-ca-cert" pod "f8f0fec9-03b7-4707-8d26-f40f5d1d312c" (UID: "f8f0fec9-03b7-4707-8d26-f40f5d1d312c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:08:35.796385 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.796363 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-certs" pod "f8f0fec9-03b7-4707-8d26-f40f5d1d312c" (UID: "f8f0fec9-03b7-4707-8d26-f40f5d1d312c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:35.798643 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.797799 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-envoy" pod "f8f0fec9-03b7-4707-8d26-f40f5d1d312c" (UID: "f8f0fec9-03b7-4707-8d26-f40f5d1d312c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:35.798643 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.797815 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-token" pod "f8f0fec9-03b7-4707-8d26-f40f5d1d312c" (UID: "f8f0fec9-03b7-4707-8d26-f40f5d1d312c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:35.798643 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.797978 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-podinfo" pod "f8f0fec9-03b7-4707-8d26-f40f5d1d312c" (UID: "f8f0fec9-03b7-4707-8d26-f40f5d1d312c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:08:35.809674 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.809645 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-kube-api-access-clnkb" pod "f8f0fec9-03b7-4707-8d26-f40f5d1d312c" (UID: "f8f0fec9-03b7-4707-8d26-f40f5d1d312c"). InnerVolumeSpecName "kube-api-access-clnkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:35.896432 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.896398 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clnkb\" (UniqueName: \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-kube-api-access-clnkb\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:35.896432 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.896427 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:35.896432 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.896440 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:35.896676 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.896449 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:35.896676 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.896458 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:35.896676 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:35.896467 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f0fec9-03b7-4707-8d26-f40f5d1d312c-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:36.121183 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.121067 4426 generic.go:356] "Generic (PLEG): container finished" podID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerID="3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef" exitCode=0 Oct 31 04:08:36.121183 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.121121 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" Oct 31 04:08:36.121183 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.121158 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" event={"ID":"f8f0fec9-03b7-4707-8d26-f40f5d1d312c","Type":"ContainerDied","Data":"3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef"} Oct 31 04:08:36.121760 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.121282 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x" event={"ID":"f8f0fec9-03b7-4707-8d26-f40f5d1d312c","Type":"ContainerDied","Data":"4afcd549d63d3bcfab98417d4f79535a3530e702ba778cbdca64eedfcd3a14bf"} Oct 31 04:08:36.121760 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.121297 4426 scope.go:122] "RemoveContainer" containerID="3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef" Oct 31 04:08:36.132811 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.132791 4426 scope.go:122] "RemoveContainer" containerID="a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760" Oct 31 04:08:36.143457 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.143437 4426 scope.go:122] "RemoveContainer" containerID="3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef" Oct 31 04:08:36.143786 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:08:36.143760 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef\": container with ID starting with 3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef not found: ID does not exist" containerID="3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef" Oct 31 04:08:36.143832 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.143801 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef"} err="failed to get container status \"3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef\": rpc error: code = NotFound desc = could not find container \"3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef\": container with ID starting with 3a5b0e6af900fb39037ae740d75be0d9ecb8f25aaa3e9440630f19c36277ddef not found: ID does not exist" Oct 31 04:08:36.143882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.143829 4426 scope.go:122] "RemoveContainer" containerID="a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760" Oct 31 04:08:36.144115 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:08:36.144092 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760\": container with ID starting with a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760 not found: ID does not exist" containerID="a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760" Oct 31 04:08:36.144188 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.144143 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760"} err="failed to get container status \"a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760\": rpc error: code = NotFound desc = could not find container \"a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760\": container with ID starting with a31077ab8b5918e89b16eec49596306a3edca552aeb5566b845d5abe3f8de760 not found: ID does not exist" Oct 31 04:08:36.272789 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.272760 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:36.280660 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.280630 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x"] Oct 31 04:08:36.325593 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.325554 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-537e34-openshift-default-57bf7cc76f-8w55x"] Oct 31 04:08:36.402426 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.402396 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-kube-api-access-bgtvk\" (UniqueName: \"kubernetes.io/projected/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-kube-api-access-bgtvk\") pod \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " Oct 31 04:08:36.402626 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.402537 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-xks-ca\") pod \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " Oct 31 04:08:36.402626 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.402562 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-openshift-ca\") pod \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\" (UID: \"41c5a965-165b-4ed6-8f4e-8d740fe1bd24\") " Oct 31 04:08:36.402963 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.402931 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-openshift-ca" pod "41c5a965-165b-4ed6-8f4e-8d740fe1bd24" (UID: "41c5a965-165b-4ed6-8f4e-8d740fe1bd24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:08:36.404532 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.404510 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-kube-api-access-bgtvk" pod "41c5a965-165b-4ed6-8f4e-8d740fe1bd24" (UID: "41c5a965-165b-4ed6-8f4e-8d740fe1bd24"). InnerVolumeSpecName "kube-api-access-bgtvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:36.404797 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.404775 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-xks-ca" pod "41c5a965-165b-4ed6-8f4e-8d740fe1bd24" (UID: "41c5a965-165b-4ed6-8f4e-8d740fe1bd24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:08:36.504257 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.504216 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgtvk\" (UniqueName: \"kubernetes.io/projected/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-kube-api-access-bgtvk\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:36.504257 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.504248 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:36.504257 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:36.504258 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c5a965-165b-4ed6-8f4e-8d740fe1bd24-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:37.129684 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:37.129650 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" path="/var/lib/kubelet/pods/f8f0fec9-03b7-4707-8d26-f40f5d1d312c/volumes" Oct 31 04:08:37.130359 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:37.130338 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" event={"ID":"41c5a965-165b-4ed6-8f4e-8d740fe1bd24","Type":"ContainerDied","Data":"14963ad101913e09b1f9420c1aeeb97612bb3fd92c9296e9df9e98b8a99741d1"} Oct 31 04:08:37.130465 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:37.130368 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14963ad101913e09b1f9420c1aeeb97612bb3fd92c9296e9df9e98b8a99741d1" Oct 31 04:08:37.130465 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:37.130378 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24" Oct 31 04:08:41.046612 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:41.046578 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp"] Oct 31 04:08:41.092837 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:41.092797 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6277d04c-mwvzp"] Oct 31 04:08:41.130645 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:41.130613 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94997f9-71c2-4670-b680-15708a3b0ff0" path="/var/lib/kubelet/pods/e94997f9-71c2-4670-b680-15708a3b0ff0/volumes" Oct 31 04:08:44.253438 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:44.253402 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb"] Oct 31 04:08:44.352626 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:44.352589 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg"] Oct 31 04:08:44.825046 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:44.825002 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf"] Oct 31 04:08:44.830965 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:44.830931 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a64fd2-9kmxf"] Oct 31 04:08:45.130730 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.130645 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7614726-c3ed-48eb-9a1b-c0554cf3fe10" path="/var/lib/kubelet/pods/b7614726-c3ed-48eb-9a1b-c0554cf3fe10/volumes" Oct 31 04:08:45.184073 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.184035 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h"] Oct 31 04:08:45.184641 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.184623 4426 cpu_manager.go:414] "containerMap: removing container" podUID="41c5a965-165b-4ed6-8f4e-8d740fe1bd24" containerName="revoke-keys" Oct 31 04:08:45.184710 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.184643 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c5a965-165b-4ed6-8f4e-8d740fe1bd24" containerName="revoke-keys" Oct 31 04:08:45.184710 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.184662 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" Oct 31 04:08:45.184710 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.184669 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" Oct 31 04:08:45.184710 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.184680 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" Oct 31 04:08:45.184710 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.184687 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" Oct 31 04:08:45.184926 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.184808 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c5a965-165b-4ed6-8f4e-8d740fe1bd24" containerName="revoke-keys" Oct 31 04:08:45.184926 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.184818 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" Oct 31 04:08:45.184926 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.184826 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f0fec9-03b7-4707-8d26-f40f5d1d312c" containerName="istio-proxy" Oct 31 04:08:45.189899 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.189876 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:45.239008 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.238977 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:08:45.239008 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.238978 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:08:45.244401 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.244370 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h"] Oct 31 04:08:45.274406 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.274374 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:08:45.274909 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.274566 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:08:45.307037 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.307001 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbz8p\" (UniqueName: \"kubernetes.io/projected/81ff872d-6d53-4e47-af49-a446110bf609-kube-api-access-gbz8p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:45.307626 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.307599 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:45.308032 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.308010 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:45.410405 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.410365 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:45.410614 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.410469 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:45.410614 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.410559 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbz8p\" (UniqueName: \"kubernetes.io/projected/81ff872d-6d53-4e47-af49-a446110bf609-kube-api-access-gbz8p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:45.411396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.411367 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:45.412795 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.412766 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:45.468002 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.467971 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbz8p\" (UniqueName: \"kubernetes.io/projected/81ff872d-6d53-4e47-af49-a446110bf609-kube-api-access-gbz8p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:45.504617 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.504587 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:45.925284 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:45.925244 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h"] Oct 31 04:08:45.929955 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:08:45.929108 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ff872d_6d53_4e47_af49_a446110bf609.slice/crio-2ef54bf04ea09343494b8876c36e2d32cb4f15c99d374e4d3541a90b64454dbf WatchSource:0}: Error finding container 2ef54bf04ea09343494b8876c36e2d32cb4f15c99d374e4d3541a90b64454dbf: Status 404 returned error can't find the container with id 2ef54bf04ea09343494b8876c36e2d32cb4f15c99d374e4d3541a90b64454dbf Oct 31 04:08:46.202942 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:46.202845 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" event={"ID":"81ff872d-6d53-4e47-af49-a446110bf609","Type":"ContainerStarted","Data":"2ef54bf04ea09343494b8876c36e2d32cb4f15c99d374e4d3541a90b64454dbf"} Oct 31 04:08:47.213239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:47.213204 4426 generic.go:356] "Generic (PLEG): container finished" podID="81ff872d-6d53-4e47-af49-a446110bf609" containerID="46c61caf67377536b8a7cceba99447f209e9821f57c7be372de6134cec2636d7" exitCode=0 Oct 31 04:08:47.213644 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:47.213281 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" event={"ID":"81ff872d-6d53-4e47-af49-a446110bf609","Type":"ContainerDied","Data":"46c61caf67377536b8a7cceba99447f209e9821f57c7be372de6134cec2636d7"} Oct 31 04:08:48.375589 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:48.375555 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:48.551492 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:48.551401 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ff872d-6d53-4e47-af49-a446110bf609-kube-api-access-gbz8p\" (UniqueName: \"kubernetes.io/projected/81ff872d-6d53-4e47-af49-a446110bf609-kube-api-access-gbz8p\") pod \"81ff872d-6d53-4e47-af49-a446110bf609\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " Oct 31 04:08:48.551492 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:48.551462 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-openshift-ca\") pod \"81ff872d-6d53-4e47-af49-a446110bf609\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " Oct 31 04:08:48.551730 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:48.551666 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-xks-ca\") pod \"81ff872d-6d53-4e47-af49-a446110bf609\" (UID: \"81ff872d-6d53-4e47-af49-a446110bf609\") " Oct 31 04:08:48.551890 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:48.551864 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-openshift-ca" pod "81ff872d-6d53-4e47-af49-a446110bf609" (UID: "81ff872d-6d53-4e47-af49-a446110bf609"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:08:48.552116 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:48.552100 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:48.553619 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:48.553587 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ff872d-6d53-4e47-af49-a446110bf609-kube-api-access-gbz8p" pod "81ff872d-6d53-4e47-af49-a446110bf609" (UID: "81ff872d-6d53-4e47-af49-a446110bf609"). InnerVolumeSpecName "kube-api-access-gbz8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:48.553619 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:48.553606 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-xks-ca" pod "81ff872d-6d53-4e47-af49-a446110bf609" (UID: "81ff872d-6d53-4e47-af49-a446110bf609"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:08:48.653669 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:48.653628 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ff872d-6d53-4e47-af49-a446110bf609-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:48.653669 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:48.653663 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbz8p\" (UniqueName: \"kubernetes.io/projected/81ff872d-6d53-4e47-af49-a446110bf609-kube-api-access-gbz8p\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:49.229637 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.229599 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" Oct 31 04:08:49.229797 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.229632 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h" event={"ID":"81ff872d-6d53-4e47-af49-a446110bf609","Type":"ContainerDied","Data":"2ef54bf04ea09343494b8876c36e2d32cb4f15c99d374e4d3541a90b64454dbf"} Oct 31 04:08:49.229797 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.229687 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ef54bf04ea09343494b8876c36e2d32cb4f15c99d374e4d3541a90b64454dbf" Oct 31 04:08:49.254788 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.254717 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" podUID="2dd1b4c2-d57e-407e-805c-85158275e778" containerName="payload-processing" containerID="cri-o://15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480" gracePeriod=25 Oct 31 04:08:49.353034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.352986 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" podUID="e9d7dd6d-4880-4dc2-b785-93c517a1d176" containerName="payload-pre-processing" containerID="cri-o://2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315" gracePeriod=25 Oct 31 04:08:49.526885 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.526854 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:08:49.627986 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.627961 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:08:49.665837 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.665793 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd1b4c2-d57e-407e-805c-85158275e778-kube-api-access-rqnq4\" (UniqueName: \"kubernetes.io/projected/2dd1b4c2-d57e-407e-805c-85158275e778-kube-api-access-rqnq4\") pod \"2dd1b4c2-d57e-407e-805c-85158275e778\" (UID: \"2dd1b4c2-d57e-407e-805c-85158275e778\") " Oct 31 04:08:49.666052 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.665906 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd1b4c2-d57e-407e-805c-85158275e778-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd1b4c2-d57e-407e-805c-85158275e778-plugins-config-volume\") pod \"2dd1b4c2-d57e-407e-805c-85158275e778\" (UID: \"2dd1b4c2-d57e-407e-805c-85158275e778\") " Oct 31 04:08:49.666410 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.666379 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd1b4c2-d57e-407e-805c-85158275e778-plugins-config-volume" pod "2dd1b4c2-d57e-407e-805c-85158275e778" (UID: "2dd1b4c2-d57e-407e-805c-85158275e778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:08:49.666612 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.666585 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd1b4c2-d57e-407e-805c-85158275e778-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:49.668239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.668215 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd1b4c2-d57e-407e-805c-85158275e778-kube-api-access-rqnq4" pod "2dd1b4c2-d57e-407e-805c-85158275e778" (UID: "2dd1b4c2-d57e-407e-805c-85158275e778"). InnerVolumeSpecName "kube-api-access-rqnq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:49.767247 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.767150 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d7dd6d-4880-4dc2-b785-93c517a1d176-kube-api-access-rf44b\" (UniqueName: \"kubernetes.io/projected/e9d7dd6d-4880-4dc2-b785-93c517a1d176-kube-api-access-rf44b\") pod \"e9d7dd6d-4880-4dc2-b785-93c517a1d176\" (UID: \"e9d7dd6d-4880-4dc2-b785-93c517a1d176\") " Oct 31 04:08:49.767247 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.767233 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d7dd6d-4880-4dc2-b785-93c517a1d176-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d7dd6d-4880-4dc2-b785-93c517a1d176-plugins-config-volume\") pod \"e9d7dd6d-4880-4dc2-b785-93c517a1d176\" (UID: \"e9d7dd6d-4880-4dc2-b785-93c517a1d176\") " Oct 31 04:08:49.767656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.767626 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d7dd6d-4880-4dc2-b785-93c517a1d176-plugins-config-volume" pod "e9d7dd6d-4880-4dc2-b785-93c517a1d176" (UID: "e9d7dd6d-4880-4dc2-b785-93c517a1d176"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:08:49.767807 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.767789 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqnq4\" (UniqueName: \"kubernetes.io/projected/2dd1b4c2-d57e-407e-805c-85158275e778-kube-api-access-rqnq4\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:49.767871 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.767814 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d7dd6d-4880-4dc2-b785-93c517a1d176-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:49.769684 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.769648 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d7dd6d-4880-4dc2-b785-93c517a1d176-kube-api-access-rf44b" pod "e9d7dd6d-4880-4dc2-b785-93c517a1d176" (UID: "e9d7dd6d-4880-4dc2-b785-93c517a1d176"). InnerVolumeSpecName "kube-api-access-rf44b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:49.868631 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:49.868592 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf44b\" (UniqueName: \"kubernetes.io/projected/e9d7dd6d-4880-4dc2-b785-93c517a1d176-kube-api-access-rf44b\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:50.241204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.241171 4426 generic.go:356] "Generic (PLEG): container finished" podID="2dd1b4c2-d57e-407e-805c-85158275e778" containerID="15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480" exitCode=0 Oct 31 04:08:50.241432 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.241240 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" Oct 31 04:08:50.241432 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.241264 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" event={"ID":"2dd1b4c2-d57e-407e-805c-85158275e778","Type":"ContainerDied","Data":"15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480"} Oct 31 04:08:50.241432 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.241329 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb" event={"ID":"2dd1b4c2-d57e-407e-805c-85158275e778","Type":"ContainerDied","Data":"c7f11be0b43f037cd487e7d5c548de2565d53ce29dbfd8c220788b3f0767581d"} Oct 31 04:08:50.241432 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.241362 4426 scope.go:122] "RemoveContainer" containerID="15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480" Oct 31 04:08:50.242942 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.242915 4426 generic.go:356] "Generic (PLEG): container finished" podID="e9d7dd6d-4880-4dc2-b785-93c517a1d176" containerID="2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315" exitCode=0 Oct 31 04:08:50.243079 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.242972 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" Oct 31 04:08:50.243079 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.243028 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" event={"ID":"e9d7dd6d-4880-4dc2-b785-93c517a1d176","Type":"ContainerDied","Data":"2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315"} Oct 31 04:08:50.243079 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.243069 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg" event={"ID":"e9d7dd6d-4880-4dc2-b785-93c517a1d176","Type":"ContainerDied","Data":"6041b4efa1fbbc1a896f89db46cc58953941a11d229a81bc41d1a5cb635973e5"} Oct 31 04:08:50.255562 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.255400 4426 scope.go:122] "RemoveContainer" containerID="15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480" Oct 31 04:08:50.260843 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:08:50.260792 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480\": container with ID starting with 15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480 not found: ID does not exist" containerID="15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480" Oct 31 04:08:50.260941 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.260832 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480"} err="failed to get container status \"15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480\": rpc error: code = NotFound desc = could not find container \"15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480\": container with ID starting with 15b2c429f8e68b3c79e344b222397a2f1fc4b0cf5fd326521cdf4d32efabd480 not found: ID does not exist" Oct 31 04:08:50.260941 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.260860 4426 scope.go:122] "RemoveContainer" containerID="2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315" Oct 31 04:08:50.272175 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.272149 4426 scope.go:122] "RemoveContainer" containerID="2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315" Oct 31 04:08:50.272513 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:08:50.272488 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315\": container with ID starting with 2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315 not found: ID does not exist" containerID="2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315" Oct 31 04:08:50.272629 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.272518 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315"} err="failed to get container status \"2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315\": rpc error: code = NotFound desc = could not find container \"2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315\": container with ID starting with 2dac51e4c7da68fa420ef7ac286623055faeb3d70dc4cc257a20630834950315 not found: ID does not exist" Oct 31 04:08:50.397496 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.397461 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg"] Oct 31 04:08:50.403374 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.403342 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3473a4-b6f688f9-m5tqg"] Oct 31 04:08:50.506455 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.506370 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb"] Oct 31 04:08:50.510352 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:50.510310 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3473a4-57b55dbf-cxdbb"] Oct 31 04:08:51.131590 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:51.130746 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd1b4c2-d57e-407e-805c-85158275e778" path="/var/lib/kubelet/pods/2dd1b4c2-d57e-407e-805c-85158275e778/volumes" Oct 31 04:08:51.132679 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:51.132162 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d7dd6d-4880-4dc2-b785-93c517a1d176" path="/var/lib/kubelet/pods/e9d7dd6d-4880-4dc2-b785-93c517a1d176/volumes" Oct 31 04:08:51.660562 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:51.660525 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8"] Oct 31 04:08:51.660855 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:51.660748 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" containerID="cri-o://f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7" gracePeriod=30 Oct 31 04:08:54.111573 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:54.111538 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:08:54.125519 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:54.125488 4426 patch_prober.go:29] interesting pod/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:08:54.125684 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:54.125554 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:08:56.941248 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:56.941174 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 04:08:57.040544 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.040505 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-data\") pod \"5a8f0271-7150-400d-bd05-da5fa011a1cf\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " Oct 31 04:08:57.040866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.040562 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-podinfo\") pod \"5a8f0271-7150-400d-bd05-da5fa011a1cf\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " Oct 31 04:08:57.040866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.040615 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-token\" (UniqueName: \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-token\") pod \"5a8f0271-7150-400d-bd05-da5fa011a1cf\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " Oct 31 04:08:57.040866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.040661 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a8f0271-7150-400d-bd05-da5fa011a1cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8f0271-7150-400d-bd05-da5fa011a1cf-istiod-ca-cert\") pod \"5a8f0271-7150-400d-bd05-da5fa011a1cf\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " Oct 31 04:08:57.040866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.040699 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-envoy\") pod \"5a8f0271-7150-400d-bd05-da5fa011a1cf\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " Oct 31 04:08:57.040866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.040735 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-certs\") pod \"5a8f0271-7150-400d-bd05-da5fa011a1cf\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " Oct 31 04:08:57.040866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.040802 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-socket\") pod \"5a8f0271-7150-400d-bd05-da5fa011a1cf\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " Oct 31 04:08:57.040866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.040829 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-kube-api-access-rkmxh\" (UniqueName: \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-kube-api-access-rkmxh\") pod \"5a8f0271-7150-400d-bd05-da5fa011a1cf\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " Oct 31 04:08:57.040866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.040850 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-data" pod "5a8f0271-7150-400d-bd05-da5fa011a1cf" (UID: "5a8f0271-7150-400d-bd05-da5fa011a1cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:57.041339 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.040879 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-credential-socket\") pod \"5a8f0271-7150-400d-bd05-da5fa011a1cf\" (UID: \"5a8f0271-7150-400d-bd05-da5fa011a1cf\") " Oct 31 04:08:57.041339 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.041120 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-socket" pod "5a8f0271-7150-400d-bd05-da5fa011a1cf" (UID: "5a8f0271-7150-400d-bd05-da5fa011a1cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:57.041339 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.041121 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8f0271-7150-400d-bd05-da5fa011a1cf-istiod-ca-cert" pod "5a8f0271-7150-400d-bd05-da5fa011a1cf" (UID: "5a8f0271-7150-400d-bd05-da5fa011a1cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:08:57.041504 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.041376 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-credential-socket" pod "5a8f0271-7150-400d-bd05-da5fa011a1cf" (UID: "5a8f0271-7150-400d-bd05-da5fa011a1cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:57.041504 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.041393 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-certs" pod "5a8f0271-7150-400d-bd05-da5fa011a1cf" (UID: "5a8f0271-7150-400d-bd05-da5fa011a1cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:57.041677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.041577 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:57.041677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.041599 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8f0271-7150-400d-bd05-da5fa011a1cf-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:57.041677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.041613 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:57.041677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.041627 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:57.041677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.041641 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:57.043727 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.043567 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-podinfo" pod "5a8f0271-7150-400d-bd05-da5fa011a1cf" (UID: "5a8f0271-7150-400d-bd05-da5fa011a1cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:08:57.043727 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.043679 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-kube-api-access-rkmxh" pod "5a8f0271-7150-400d-bd05-da5fa011a1cf" (UID: "5a8f0271-7150-400d-bd05-da5fa011a1cf"). InnerVolumeSpecName "kube-api-access-rkmxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:57.043919 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.043853 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-envoy" pod "5a8f0271-7150-400d-bd05-da5fa011a1cf" (UID: "5a8f0271-7150-400d-bd05-da5fa011a1cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:57.043919 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.043865 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-token" pod "5a8f0271-7150-400d-bd05-da5fa011a1cf" (UID: "5a8f0271-7150-400d-bd05-da5fa011a1cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:57.143468 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.143298 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:57.143468 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.143328 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:57.143468 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.143343 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8f0271-7150-400d-bd05-da5fa011a1cf-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:57.143468 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.143358 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkmxh\" (UniqueName: \"kubernetes.io/projected/5a8f0271-7150-400d-bd05-da5fa011a1cf-kube-api-access-rkmxh\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:57.307287 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.307196 4426 generic.go:356] "Generic (PLEG): container finished" podID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerID="f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7" exitCode=0 Oct 31 04:08:57.307287 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.307250 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" Oct 31 04:08:57.307287 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.307260 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" event={"ID":"5a8f0271-7150-400d-bd05-da5fa011a1cf","Type":"ContainerDied","Data":"f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7"} Oct 31 04:08:57.307519 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.307306 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8" event={"ID":"5a8f0271-7150-400d-bd05-da5fa011a1cf","Type":"ContainerDied","Data":"320f57cc75ec4ffe5e2a50cd19ff6cbf9c222d55c22aa35652c01891865668d7"} Oct 31 04:08:57.307519 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.307328 4426 scope.go:122] "RemoveContainer" containerID="f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7" Oct 31 04:08:57.318022 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.317996 4426 scope.go:122] "RemoveContainer" containerID="991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889" Oct 31 04:08:57.328273 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.328250 4426 scope.go:122] "RemoveContainer" containerID="f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7" Oct 31 04:08:57.328583 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:08:57.328563 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7\": container with ID starting with f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7 not found: ID does not exist" containerID="f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7" Oct 31 04:08:57.328666 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.328593 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7"} err="failed to get container status \"f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7\": rpc error: code = NotFound desc = could not find container \"f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7\": container with ID starting with f8f3dc6b89b063bf0e180f72556d84b5ce330795754ffbeaef261d439a124aa7 not found: ID does not exist" Oct 31 04:08:57.328666 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.328612 4426 scope.go:122] "RemoveContainer" containerID="991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889" Oct 31 04:08:57.328872 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:08:57.328855 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889\": container with ID starting with 991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889 not found: ID does not exist" containerID="991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889" Oct 31 04:08:57.328940 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.328876 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889"} err="failed to get container status \"991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889\": rpc error: code = NotFound desc = could not find container \"991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889\": container with ID starting with 991078bbc4812a378bbed35e500f1c90fb6ab681115d88122b849ef347a40889 not found: ID does not exist" Oct 31 04:08:57.464221 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.464181 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8"] Oct 31 04:08:57.470152 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:57.470105 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a64fd2-openshift-default-694cc85679-qfjb8"] Oct 31 04:08:59.130556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:08:59.130519 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" path="/var/lib/kubelet/pods/5a8f0271-7150-400d-bd05-da5fa011a1cf/volumes" Oct 31 04:09:02.846711 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:09:02.846672 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:09:04.049536 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:04.049498 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h"] Oct 31 04:09:04.095628 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:04.095581 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5a5baa-lcz5h"] Oct 31 04:09:05.129931 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.129893 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ff872d-6d53-4e47-af49-a446110bf609" path="/var/lib/kubelet/pods/81ff872d-6d53-4e47-af49-a446110bf609/volumes" Oct 31 04:09:05.906492 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.906447 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6968-ff11b1"] Oct 31 04:09:05.907309 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907285 4426 cpu_manager.go:414] "containerMap: removing container" podUID="2dd1b4c2-d57e-407e-805c-85158275e778" containerName="payload-processing" Oct 31 04:09:05.907309 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907309 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd1b4c2-d57e-407e-805c-85158275e778" containerName="payload-processing" Oct 31 04:09:05.907509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907331 4426 cpu_manager.go:414] "containerMap: removing container" podUID="81ff872d-6d53-4e47-af49-a446110bf609" containerName="revoke-keys" Oct 31 04:09:05.907509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907339 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ff872d-6d53-4e47-af49-a446110bf609" containerName="revoke-keys" Oct 31 04:09:05.907509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907363 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" Oct 31 04:09:05.907509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907371 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" Oct 31 04:09:05.907509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907397 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" Oct 31 04:09:05.907509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907403 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" Oct 31 04:09:05.907509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907427 4426 cpu_manager.go:414] "containerMap: removing container" podUID="e9d7dd6d-4880-4dc2-b785-93c517a1d176" containerName="payload-pre-processing" Oct 31 04:09:05.907509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907434 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d7dd6d-4880-4dc2-b785-93c517a1d176" containerName="payload-pre-processing" Oct 31 04:09:05.907922 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907550 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" Oct 31 04:09:05.907922 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907561 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d7dd6d-4880-4dc2-b785-93c517a1d176" containerName="payload-pre-processing" Oct 31 04:09:05.907922 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907574 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd1b4c2-d57e-407e-805c-85158275e778" containerName="payload-processing" Oct 31 04:09:05.907922 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907583 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ff872d-6d53-4e47-af49-a446110bf609" containerName="revoke-keys" Oct 31 04:09:05.907922 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.907594 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8f0271-7150-400d-bd05-da5fa011a1cf" containerName="istio-proxy" Oct 31 04:09:05.913238 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.913211 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ff11b1" Oct 31 04:09:05.965658 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:05.963398 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-ff11b1"] Oct 31 04:09:06.018342 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:06.018308 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2x2q5\"" Oct 31 04:09:06.056145 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:06.056096 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g674\" (UniqueName: \"kubernetes.io/projected/b4528afc-e74e-406a-9ed5-b2a872648721-kube-api-access-5g674\") pod \"test-curl-6968-ff11b1\" (UID: \"b4528afc-e74e-406a-9ed5-b2a872648721\") " pod="opendatahub/test-curl-6968-ff11b1" Oct 31 04:09:06.157864 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:06.157758 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g674\" (UniqueName: \"kubernetes.io/projected/b4528afc-e74e-406a-9ed5-b2a872648721-kube-api-access-5g674\") pod \"test-curl-6968-ff11b1\" (UID: \"b4528afc-e74e-406a-9ed5-b2a872648721\") " pod="opendatahub/test-curl-6968-ff11b1" Oct 31 04:09:06.217417 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:06.217379 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g674\" (UniqueName: \"kubernetes.io/projected/b4528afc-e74e-406a-9ed5-b2a872648721-kube-api-access-5g674\") pod \"test-curl-6968-ff11b1\" (UID: \"b4528afc-e74e-406a-9ed5-b2a872648721\") " pod="opendatahub/test-curl-6968-ff11b1" Oct 31 04:09:06.230674 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:06.230638 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ff11b1" Oct 31 04:09:06.611935 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:06.611848 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-ff11b1"] Oct 31 04:09:06.616974 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:09:06.616945 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4528afc_e74e_406a_9ed5_b2a872648721.slice/crio-29b37053f868efd7f8584c1f45ddc74c162d9be28cf4e97e35a5bbc2d9613247 WatchSource:0}: Error finding container 29b37053f868efd7f8584c1f45ddc74c162d9be28cf4e97e35a5bbc2d9613247: Status 404 returned error can't find the container with id 29b37053f868efd7f8584c1f45ddc74c162d9be28cf4e97e35a5bbc2d9613247 Oct 31 04:09:07.384053 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:07.384011 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-ff11b1" event={"ID":"b4528afc-e74e-406a-9ed5-b2a872648721","Type":"ContainerStarted","Data":"f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483"} Oct 31 04:09:07.384473 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:07.384067 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-ff11b1" event={"ID":"b4528afc-e74e-406a-9ed5-b2a872648721","Type":"ContainerStarted","Data":"29b37053f868efd7f8584c1f45ddc74c162d9be28cf4e97e35a5bbc2d9613247"} Oct 31 04:09:07.493174 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:07.493106 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6968-ff11b1" podStartSLOduration=2.493093086 podStartE2EDuration="2.493093086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:07.489674222 +0000 UTC m=+2444.490299451" watchObservedRunningTime="2026-10-31 04:09:07.493093086 +0000 UTC m=+2444.493718313" Oct 31 04:09:08.924741 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:08.924701 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6968-ff11b1"] Oct 31 04:09:08.929476 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:08.929436 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6968-ff11b1"] Oct 31 04:09:09.398291 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.398239 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6968-ff11b1" podUID="b4528afc-e74e-406a-9ed5-b2a872648721" containerName="test-curl-6968-ff11b1" containerID="cri-o://f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483" gracePeriod=2 Oct 31 04:09:09.796354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.796257 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q"] Oct 31 04:09:09.796812 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.796797 4426 cpu_manager.go:414] "containerMap: removing container" podUID="b4528afc-e74e-406a-9ed5-b2a872648721" containerName="test-curl-6968-ff11b1" Oct 31 04:09:09.796895 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.796812 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4528afc-e74e-406a-9ed5-b2a872648721" containerName="test-curl-6968-ff11b1" Oct 31 04:09:09.796956 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.796918 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4528afc-e74e-406a-9ed5-b2a872648721" containerName="test-curl-6968-ff11b1" Oct 31 04:09:09.800681 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.800656 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:09.895840 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.895804 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:09.896022 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.895932 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:09.896079 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.896062 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45lsl\" (UniqueName: \"kubernetes.io/projected/c4d2bc51-8084-4bb7-9366-801d380a6807-kube-api-access-45lsl\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:09.896771 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.896746 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:09:09.896857 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.896751 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:09:09.896857 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.896779 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:09:09.896857 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.896783 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:09:09.900589 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.900570 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q"] Oct 31 04:09:09.997411 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.997376 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lsl\" (UniqueName: \"kubernetes.io/projected/c4d2bc51-8084-4bb7-9366-801d380a6807-kube-api-access-45lsl\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:09.997893 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.997422 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:09.997893 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.997479 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:09.998156 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.998114 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:09.999814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:09.999790 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:10.055820 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:10.055735 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lsl\" (UniqueName: \"kubernetes.io/projected/c4d2bc51-8084-4bb7-9366-801d380a6807-kube-api-access-45lsl\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:10.114919 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:10.114886 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:10.502214 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:10.502174 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q"] Oct 31 04:09:10.505947 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:09:10.505917 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d2bc51_8084_4bb7_9366_801d380a6807.slice/crio-8970a3be9978e776b6f4dee585a5394dbe9fe2d6e811481671c3d6e688812a3b WatchSource:0}: Error finding container 8970a3be9978e776b6f4dee585a5394dbe9fe2d6e811481671c3d6e688812a3b: Status 404 returned error can't find the container with id 8970a3be9978e776b6f4dee585a5394dbe9fe2d6e811481671c3d6e688812a3b Oct 31 04:09:11.415090 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:11.415058 4426 generic.go:356] "Generic (PLEG): container finished" podID="c4d2bc51-8084-4bb7-9366-801d380a6807" containerID="e5021bb84e992e7ac4579630074d190b2dfc851a683e5ace44d4ac290bced67b" exitCode=0 Oct 31 04:09:11.415612 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:11.415155 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" event={"ID":"c4d2bc51-8084-4bb7-9366-801d380a6807","Type":"ContainerDied","Data":"e5021bb84e992e7ac4579630074d190b2dfc851a683e5ace44d4ac290bced67b"} Oct 31 04:09:11.415612 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:11.415208 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" event={"ID":"c4d2bc51-8084-4bb7-9366-801d380a6807","Type":"ContainerStarted","Data":"8970a3be9978e776b6f4dee585a5394dbe9fe2d6e811481671c3d6e688812a3b"} Oct 31 04:09:11.634667 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:11.634591 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ff11b1" Oct 31 04:09:11.716093 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:11.716051 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4528afc-e74e-406a-9ed5-b2a872648721-kube-api-access-5g674\" (UniqueName: \"kubernetes.io/projected/b4528afc-e74e-406a-9ed5-b2a872648721-kube-api-access-5g674\") pod \"b4528afc-e74e-406a-9ed5-b2a872648721\" (UID: \"b4528afc-e74e-406a-9ed5-b2a872648721\") " Oct 31 04:09:11.718235 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:11.718205 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4528afc-e74e-406a-9ed5-b2a872648721-kube-api-access-5g674" pod "b4528afc-e74e-406a-9ed5-b2a872648721" (UID: "b4528afc-e74e-406a-9ed5-b2a872648721"). InnerVolumeSpecName "kube-api-access-5g674". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:11.816999 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:11.816956 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g674\" (UniqueName: \"kubernetes.io/projected/b4528afc-e74e-406a-9ed5-b2a872648721-kube-api-access-5g674\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:12.430470 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.430425 4426 generic.go:356] "Generic (PLEG): container finished" podID="b4528afc-e74e-406a-9ed5-b2a872648721" containerID="f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483" exitCode=137 Oct 31 04:09:12.430987 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.430488 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ff11b1" Oct 31 04:09:12.430987 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.430616 4426 scope.go:122] "RemoveContainer" containerID="f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483" Oct 31 04:09:12.444185 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.444115 4426 scope.go:122] "RemoveContainer" containerID="f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483" Oct 31 04:09:12.444544 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:09:12.444519 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483\": container with ID starting with f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483 not found: ID does not exist" containerID="f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483" Oct 31 04:09:12.444637 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.444555 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483"} err="failed to get container status \"f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483\": rpc error: code = NotFound desc = could not find container \"f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483\": container with ID starting with f4b2f46a54f49d793045251fc014899bda8090486a580a1e80d9444a0f579483 not found: ID does not exist" Oct 31 04:09:12.495695 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:09:12.495655 4426 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-ff11b1\" is forbidden: User \"system:node:ip-10-0-134-106.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-106.us-west-2.compute.internal' and this object" podUID="b4528afc-e74e-406a-9ed5-b2a872648721" pod="opendatahub/test-curl-6968-ff11b1" Oct 31 04:09:12.590653 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.590622 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:12.638518 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:09:12.638477 4426 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-ff11b1\" is forbidden: User \"system:node:ip-10-0-134-106.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-106.us-west-2.compute.internal' and this object" podUID="b4528afc-e74e-406a-9ed5-b2a872648721" pod="opendatahub/test-curl-6968-ff11b1" Oct 31 04:09:12.732386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.732295 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d2bc51-8084-4bb7-9366-801d380a6807-kube-api-access-45lsl\" (UniqueName: \"kubernetes.io/projected/c4d2bc51-8084-4bb7-9366-801d380a6807-kube-api-access-45lsl\") pod \"c4d2bc51-8084-4bb7-9366-801d380a6807\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " Oct 31 04:09:12.732550 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.732510 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-openshift-ca\") pod \"c4d2bc51-8084-4bb7-9366-801d380a6807\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " Oct 31 04:09:12.732641 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.732624 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-xks-ca\") pod \"c4d2bc51-8084-4bb7-9366-801d380a6807\" (UID: \"c4d2bc51-8084-4bb7-9366-801d380a6807\") " Oct 31 04:09:12.732903 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.732878 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-openshift-ca" pod "c4d2bc51-8084-4bb7-9366-801d380a6807" (UID: "c4d2bc51-8084-4bb7-9366-801d380a6807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:12.733030 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.733013 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:12.734537 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.734506 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d2bc51-8084-4bb7-9366-801d380a6807-kube-api-access-45lsl" pod "c4d2bc51-8084-4bb7-9366-801d380a6807" (UID: "c4d2bc51-8084-4bb7-9366-801d380a6807"). InnerVolumeSpecName "kube-api-access-45lsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:12.734706 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.734683 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-xks-ca" pod "c4d2bc51-8084-4bb7-9366-801d380a6807" (UID: "c4d2bc51-8084-4bb7-9366-801d380a6807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:09:12.834631 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.834574 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d2bc51-8084-4bb7-9366-801d380a6807-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:12.834631 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:12.834628 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45lsl\" (UniqueName: \"kubernetes.io/projected/c4d2bc51-8084-4bb7-9366-801d380a6807-kube-api-access-45lsl\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:13.130619 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:13.130540 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4528afc-e74e-406a-9ed5-b2a872648721" path="/var/lib/kubelet/pods/b4528afc-e74e-406a-9ed5-b2a872648721/volumes" Oct 31 04:09:13.174619 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:09:13.174583 4426 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-ff11b1\" is forbidden: User \"system:node:ip-10-0-134-106.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-106.us-west-2.compute.internal' and this object" podUID="b4528afc-e74e-406a-9ed5-b2a872648721" pod="opendatahub/test-curl-6968-ff11b1" Oct 31 04:09:13.439635 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:13.439597 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" event={"ID":"c4d2bc51-8084-4bb7-9366-801d380a6807","Type":"ContainerDied","Data":"8970a3be9978e776b6f4dee585a5394dbe9fe2d6e811481671c3d6e688812a3b"} Oct 31 04:09:13.440075 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:13.439662 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8970a3be9978e776b6f4dee585a5394dbe9fe2d6e811481671c3d6e688812a3b" Oct 31 04:09:13.440075 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:13.439613 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q" Oct 31 04:09:21.194497 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.194459 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc"] Oct 31 04:09:21.295862 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.295831 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v"] Oct 31 04:09:21.669073 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.669040 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t"] Oct 31 04:09:21.669624 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.669610 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c4d2bc51-8084-4bb7-9366-801d380a6807" containerName="revoke-keys" Oct 31 04:09:21.669683 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.669625 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d2bc51-8084-4bb7-9366-801d380a6807" containerName="revoke-keys" Oct 31 04:09:21.669738 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.669730 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d2bc51-8084-4bb7-9366-801d380a6807" containerName="revoke-keys" Oct 31 04:09:21.674650 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.674621 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:21.724877 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.724839 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:09:21.725066 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.724922 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:09:21.732718 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.732681 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t"] Oct 31 04:09:21.739262 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.739231 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:09:21.739402 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.739237 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:09:21.838111 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.838074 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llcf9\" (UniqueName: \"kubernetes.io/projected/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-kube-api-access-llcf9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:21.838406 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.838309 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:21.838406 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.838366 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:21.939773 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.939680 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:21.939773 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.939735 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:21.940027 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.939780 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llcf9\" (UniqueName: \"kubernetes.io/projected/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-kube-api-access-llcf9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:21.940533 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.940508 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:21.942177 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.942151 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:21.997744 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:21.997711 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llcf9\" (UniqueName: \"kubernetes.io/projected/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-kube-api-access-llcf9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:22.009295 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:22.009265 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:22.384710 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:22.384610 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t"] Oct 31 04:09:22.388867 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:09:22.388838 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb712b2_a972_4d26_966c_3ca3eb8b9da8.slice/crio-d3858d7c38ce6e6bb5268adcb1f0cceb03c638fe00bb334b0f95bad16e857c33 WatchSource:0}: Error finding container d3858d7c38ce6e6bb5268adcb1f0cceb03c638fe00bb334b0f95bad16e857c33: Status 404 returned error can't find the container with id d3858d7c38ce6e6bb5268adcb1f0cceb03c638fe00bb334b0f95bad16e857c33 Oct 31 04:09:22.514119 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:22.514082 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" event={"ID":"ebb712b2-a972-4d26-966c-3ca3eb8b9da8","Type":"ContainerStarted","Data":"d3858d7c38ce6e6bb5268adcb1f0cceb03c638fe00bb334b0f95bad16e857c33"} Oct 31 04:09:23.523282 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:23.523247 4426 generic.go:356] "Generic (PLEG): container finished" podID="ebb712b2-a972-4d26-966c-3ca3eb8b9da8" containerID="4473dae02e8a3cb40421b77e3de5548c6c51bd6f6c8234f0ca1c64f205c2d538" exitCode=0 Oct 31 04:09:23.523763 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:23.523339 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" event={"ID":"ebb712b2-a972-4d26-966c-3ca3eb8b9da8","Type":"ContainerDied","Data":"4473dae02e8a3cb40421b77e3de5548c6c51bd6f6c8234f0ca1c64f205c2d538"} Oct 31 04:09:24.686943 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:24.686916 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:24.775137 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:24.775099 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-kube-api-access-llcf9\" (UniqueName: \"kubernetes.io/projected/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-kube-api-access-llcf9\") pod \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " Oct 31 04:09:24.775345 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:24.775196 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-openshift-ca\") pod \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " Oct 31 04:09:24.775345 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:24.775274 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-xks-ca\") pod \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\" (UID: \"ebb712b2-a972-4d26-966c-3ca3eb8b9da8\") " Oct 31 04:09:24.775568 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:24.775543 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-openshift-ca" pod "ebb712b2-a972-4d26-966c-3ca3eb8b9da8" (UID: "ebb712b2-a972-4d26-966c-3ca3eb8b9da8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:24.775758 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:24.775736 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:24.777409 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:24.777386 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-kube-api-access-llcf9" pod "ebb712b2-a972-4d26-966c-3ca3eb8b9da8" (UID: "ebb712b2-a972-4d26-966c-3ca3eb8b9da8"). InnerVolumeSpecName "kube-api-access-llcf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:24.777482 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:24.777412 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-xks-ca" pod "ebb712b2-a972-4d26-966c-3ca3eb8b9da8" (UID: "ebb712b2-a972-4d26-966c-3ca3eb8b9da8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:09:24.876941 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:24.876850 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llcf9\" (UniqueName: \"kubernetes.io/projected/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-kube-api-access-llcf9\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:24.876941 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:24.876884 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb712b2-a972-4d26-966c-3ca3eb8b9da8-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:25.543743 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:25.543627 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" Oct 31 04:09:25.543743 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:25.543657 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t" event={"ID":"ebb712b2-a972-4d26-966c-3ca3eb8b9da8","Type":"ContainerDied","Data":"d3858d7c38ce6e6bb5268adcb1f0cceb03c638fe00bb334b0f95bad16e857c33"} Oct 31 04:09:25.543743 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:25.543716 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3858d7c38ce6e6bb5268adcb1f0cceb03c638fe00bb334b0f95bad16e857c33" Oct 31 04:09:26.020899 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:26.020859 4426 scope.go:122] "RemoveContainer" containerID="2e63bf5a33f4e057dd749747e93e6e5a0be7187b163fd138eb5f5c5f686c5e8d" Oct 31 04:09:26.032918 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:26.032892 4426 scope.go:122] "RemoveContainer" containerID="49cdca9556969fa5823846ede5b6ee5e05136791787cd896f414cdc6d3c11c26" Oct 31 04:09:26.195467 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:26.195424 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" podUID="3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e" containerName="payload-processing" containerID="cri-o://1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522" gracePeriod=25 Oct 31 04:09:26.296678 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:26.296595 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" podUID="0182921c-a48d-47db-a101-5cd137abf755" containerName="payload-pre-processing" containerID="cri-o://5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc" gracePeriod=25 Oct 31 04:09:28.813626 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:28.813591 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:09:29.351181 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:29.351115 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" podUID="0182921c-a48d-47db-a101-5cd137abf755" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:09:29.351400 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:29.351196 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" podUID="3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:09:32.855902 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:09:32.855861 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:09:32.961938 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:32.961901 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24"] Oct 31 04:09:33.008202 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:33.008160 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3473a4-qwc24"] Oct 31 04:09:33.099540 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:33.099509 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw"] Oct 31 04:09:33.129171 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:33.129075 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c5a965-165b-4ed6-8f4e-8d740fe1bd24" path="/var/lib/kubelet/pods/41c5a965-165b-4ed6-8f4e-8d740fe1bd24/volumes" Oct 31 04:09:33.197884 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:33.197846 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n"] Oct 31 04:09:38.099930 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.099821 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" podUID="96e43cfe-4f33-4011-8588-eae8270d23cd" containerName="payload-processing" containerID="cri-o://348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce" gracePeriod=25 Oct 31 04:09:38.199038 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.198964 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" podUID="c653c61a-7630-46e7-99e3-f0b48471b6a1" containerName="payload-pre-processing" containerID="cri-o://0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec" gracePeriod=25 Oct 31 04:09:38.382861 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.382828 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:09:38.422690 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.422638 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e43cfe-4f33-4011-8588-eae8270d23cd-kube-api-access-5hqgn\" (UniqueName: \"kubernetes.io/projected/96e43cfe-4f33-4011-8588-eae8270d23cd-kube-api-access-5hqgn\") pod \"96e43cfe-4f33-4011-8588-eae8270d23cd\" (UID: \"96e43cfe-4f33-4011-8588-eae8270d23cd\") " Oct 31 04:09:38.422903 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.422776 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e43cfe-4f33-4011-8588-eae8270d23cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e43cfe-4f33-4011-8588-eae8270d23cd-plugins-config-volume\") pod \"96e43cfe-4f33-4011-8588-eae8270d23cd\" (UID: \"96e43cfe-4f33-4011-8588-eae8270d23cd\") " Oct 31 04:09:38.423441 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.423413 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e43cfe-4f33-4011-8588-eae8270d23cd-plugins-config-volume" pod "96e43cfe-4f33-4011-8588-eae8270d23cd" (UID: "96e43cfe-4f33-4011-8588-eae8270d23cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:38.424953 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.424927 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e43cfe-4f33-4011-8588-eae8270d23cd-kube-api-access-5hqgn" pod "96e43cfe-4f33-4011-8588-eae8270d23cd" (UID: "96e43cfe-4f33-4011-8588-eae8270d23cd"). InnerVolumeSpecName "kube-api-access-5hqgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:38.487302 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.487277 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:09:38.523634 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.523596 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c653c61a-7630-46e7-99e3-f0b48471b6a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c653c61a-7630-46e7-99e3-f0b48471b6a1-plugins-config-volume\") pod \"c653c61a-7630-46e7-99e3-f0b48471b6a1\" (UID: \"c653c61a-7630-46e7-99e3-f0b48471b6a1\") " Oct 31 04:09:38.523834 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.523687 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c653c61a-7630-46e7-99e3-f0b48471b6a1-kube-api-access-xxvd5\" (UniqueName: \"kubernetes.io/projected/c653c61a-7630-46e7-99e3-f0b48471b6a1-kube-api-access-xxvd5\") pod \"c653c61a-7630-46e7-99e3-f0b48471b6a1\" (UID: \"c653c61a-7630-46e7-99e3-f0b48471b6a1\") " Oct 31 04:09:38.524023 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.523998 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c653c61a-7630-46e7-99e3-f0b48471b6a1-plugins-config-volume" pod "c653c61a-7630-46e7-99e3-f0b48471b6a1" (UID: "c653c61a-7630-46e7-99e3-f0b48471b6a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:38.524216 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.524196 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e43cfe-4f33-4011-8588-eae8270d23cd-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:38.524298 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.524221 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c653c61a-7630-46e7-99e3-f0b48471b6a1-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:38.524298 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.524235 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hqgn\" (UniqueName: \"kubernetes.io/projected/96e43cfe-4f33-4011-8588-eae8270d23cd-kube-api-access-5hqgn\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:38.526489 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.526463 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c653c61a-7630-46e7-99e3-f0b48471b6a1-kube-api-access-xxvd5" pod "c653c61a-7630-46e7-99e3-f0b48471b6a1" (UID: "c653c61a-7630-46e7-99e3-f0b48471b6a1"). InnerVolumeSpecName "kube-api-access-xxvd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:38.625065 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.624974 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxvd5\" (UniqueName: \"kubernetes.io/projected/c653c61a-7630-46e7-99e3-f0b48471b6a1-kube-api-access-xxvd5\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:38.676040 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.676005 4426 generic.go:356] "Generic (PLEG): container finished" podID="c653c61a-7630-46e7-99e3-f0b48471b6a1" containerID="0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec" exitCode=0 Oct 31 04:09:38.676281 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.676048 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" Oct 31 04:09:38.676281 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.676081 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" event={"ID":"c653c61a-7630-46e7-99e3-f0b48471b6a1","Type":"ContainerDied","Data":"0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec"} Oct 31 04:09:38.676281 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.676173 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n" event={"ID":"c653c61a-7630-46e7-99e3-f0b48471b6a1","Type":"ContainerDied","Data":"772ab12ff51a122b93aa60a79db1052c6edb3fbcf49350ca4ba9be34a86426e8"} Oct 31 04:09:38.676281 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.676203 4426 scope.go:122] "RemoveContainer" containerID="0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec" Oct 31 04:09:38.677772 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.677749 4426 generic.go:356] "Generic (PLEG): container finished" podID="96e43cfe-4f33-4011-8588-eae8270d23cd" containerID="348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce" exitCode=0 Oct 31 04:09:38.677890 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.677794 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" Oct 31 04:09:38.677890 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.677833 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" event={"ID":"96e43cfe-4f33-4011-8588-eae8270d23cd","Type":"ContainerDied","Data":"348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce"} Oct 31 04:09:38.677890 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.677881 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw" event={"ID":"96e43cfe-4f33-4011-8588-eae8270d23cd","Type":"ContainerDied","Data":"83fbc7232b46a48f33b4b86518e948a4f7a869cd75852574a44760ffee6f6c74"} Oct 31 04:09:38.687045 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.687025 4426 scope.go:122] "RemoveContainer" containerID="0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec" Oct 31 04:09:38.687398 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:09:38.687373 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec\": container with ID starting with 0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec not found: ID does not exist" containerID="0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec" Oct 31 04:09:38.687510 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.687404 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec"} err="failed to get container status \"0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec\": rpc error: code = NotFound desc = could not find container \"0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec\": container with ID starting with 0476c0a8b01643c99a3b4c4308ec79e538514f411166f53310f1fbf680bd86ec not found: ID does not exist" Oct 31 04:09:38.687510 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.687421 4426 scope.go:122] "RemoveContainer" containerID="348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce" Oct 31 04:09:38.697285 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.697268 4426 scope.go:122] "RemoveContainer" containerID="348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce" Oct 31 04:09:38.697569 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:09:38.697552 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce\": container with ID starting with 348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce not found: ID does not exist" containerID="348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce" Oct 31 04:09:38.697611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.697579 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce"} err="failed to get container status \"348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce\": rpc error: code = NotFound desc = could not find container \"348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce\": container with ID starting with 348b84a15643ff4ff340df04f6155bc782373ca53681919272643b00860276ce not found: ID does not exist" Oct 31 04:09:38.833655 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.833617 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n"] Oct 31 04:09:38.838390 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.838347 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-288b63-6dd6c97c87-nbz5n"] Oct 31 04:09:38.942492 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.942440 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw"] Oct 31 04:09:38.945764 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:38.945732 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-288b63-7fdf848bd9-9dztw"] Oct 31 04:09:39.130316 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:39.130280 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e43cfe-4f33-4011-8588-eae8270d23cd" path="/var/lib/kubelet/pods/96e43cfe-4f33-4011-8588-eae8270d23cd/volumes" Oct 31 04:09:39.130812 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:39.130790 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c653c61a-7630-46e7-99e3-f0b48471b6a1" path="/var/lib/kubelet/pods/c653c61a-7630-46e7-99e3-f0b48471b6a1/volumes" Oct 31 04:09:39.351428 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:39.351334 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" podUID="0182921c-a48d-47db-a101-5cd137abf755" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:09:39.351626 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:39.351334 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" podUID="3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:09:41.281756 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.281720 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv"] Oct 31 04:09:41.282516 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.282492 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c653c61a-7630-46e7-99e3-f0b48471b6a1" containerName="payload-pre-processing" Oct 31 04:09:41.282516 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.282516 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c653c61a-7630-46e7-99e3-f0b48471b6a1" containerName="payload-pre-processing" Oct 31 04:09:41.282695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.282538 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ebb712b2-a972-4d26-966c-3ca3eb8b9da8" containerName="revoke-keys" Oct 31 04:09:41.282695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.282545 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb712b2-a972-4d26-966c-3ca3eb8b9da8" containerName="revoke-keys" Oct 31 04:09:41.282695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.282585 4426 cpu_manager.go:414] "containerMap: removing container" podUID="96e43cfe-4f33-4011-8588-eae8270d23cd" containerName="payload-processing" Oct 31 04:09:41.282695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.282593 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e43cfe-4f33-4011-8588-eae8270d23cd" containerName="payload-processing" Oct 31 04:09:41.282911 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.282743 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e43cfe-4f33-4011-8588-eae8270d23cd" containerName="payload-processing" Oct 31 04:09:41.282911 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.282763 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c653c61a-7630-46e7-99e3-f0b48471b6a1" containerName="payload-pre-processing" Oct 31 04:09:41.282911 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.282775 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb712b2-a972-4d26-966c-3ca3eb8b9da8" containerName="revoke-keys" Oct 31 04:09:41.287922 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.287900 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:41.341385 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.341347 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv"] Oct 31 04:09:41.360085 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.360048 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tnf7\" (UniqueName: \"kubernetes.io/projected/7a456a46-63ad-4621-b430-a9980f6d5c0f-kube-api-access-9tnf7\") pod \"maas-api-revoke-sub-d7b4432d-khtvv\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:41.360396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.360368 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b4432d-khtvv\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:41.360536 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.360420 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b4432d-khtvv\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:41.382487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.382454 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:09:41.382487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.382484 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:09:41.382739 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.382494 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:09:41.383883 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.383863 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:09:41.461623 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.461586 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b4432d-khtvv\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:41.461838 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.461701 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tnf7\" (UniqueName: \"kubernetes.io/projected/7a456a46-63ad-4621-b430-a9980f6d5c0f-kube-api-access-9tnf7\") pod \"maas-api-revoke-sub-d7b4432d-khtvv\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:41.461838 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.461734 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b4432d-khtvv\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:41.462403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.462383 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b4432d-khtvv\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:41.464030 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.464007 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b4432d-khtvv\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:41.519389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.519358 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tnf7\" (UniqueName: \"kubernetes.io/projected/7a456a46-63ad-4621-b430-a9980f6d5c0f-kube-api-access-9tnf7\") pod \"maas-api-revoke-sub-d7b4432d-khtvv\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:41.601883 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.601796 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:41.980018 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:41.979985 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv"] Oct 31 04:09:41.984192 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:09:41.984159 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a456a46_63ad_4621_b430_a9980f6d5c0f.slice/crio-89bc2cb272971291ce458ecf420fee691944238c47563ab2fcc743927f3113b7 WatchSource:0}: Error finding container 89bc2cb272971291ce458ecf420fee691944238c47563ab2fcc743927f3113b7: Status 404 returned error can't find the container with id 89bc2cb272971291ce458ecf420fee691944238c47563ab2fcc743927f3113b7 Oct 31 04:09:42.720678 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:42.720631 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" event={"ID":"7a456a46-63ad-4621-b430-a9980f6d5c0f","Type":"ContainerStarted","Data":"4d7888889e76138a31be7ded685a61c628bd2fbe79bc3cb928a43bec820672ac"} Oct 31 04:09:42.721162 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:42.720758 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" event={"ID":"7a456a46-63ad-4621-b430-a9980f6d5c0f","Type":"ContainerStarted","Data":"89bc2cb272971291ce458ecf420fee691944238c47563ab2fcc743927f3113b7"} Oct 31 04:09:42.831824 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:42.831779 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" podStartSLOduration=1.8317656530000002 podStartE2EDuration="1.831765653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:42.831461504 +0000 UTC m=+2479.832086732" watchObservedRunningTime="2026-10-31 04:09:42.831765653 +0000 UTC m=+2479.832390881" Oct 31 04:09:43.731817 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:43.731778 4426 generic.go:356] "Generic (PLEG): container finished" podID="7a456a46-63ad-4621-b430-a9980f6d5c0f" containerID="4d7888889e76138a31be7ded685a61c628bd2fbe79bc3cb928a43bec820672ac" exitCode=0 Oct 31 04:09:43.732389 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:43.731835 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" event={"ID":"7a456a46-63ad-4621-b430-a9980f6d5c0f","Type":"ContainerDied","Data":"4d7888889e76138a31be7ded685a61c628bd2fbe79bc3cb928a43bec820672ac"} Oct 31 04:09:44.901604 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:44.901571 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:44.995515 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:44.995427 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-xks-ca\") pod \"7a456a46-63ad-4621-b430-a9980f6d5c0f\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " Oct 31 04:09:44.995728 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:44.995556 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-openshift-ca\") pod \"7a456a46-63ad-4621-b430-a9980f6d5c0f\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " Oct 31 04:09:44.995728 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:44.995647 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a456a46-63ad-4621-b430-a9980f6d5c0f-kube-api-access-9tnf7\" (UniqueName: \"kubernetes.io/projected/7a456a46-63ad-4621-b430-a9980f6d5c0f-kube-api-access-9tnf7\") pod \"7a456a46-63ad-4621-b430-a9980f6d5c0f\" (UID: \"7a456a46-63ad-4621-b430-a9980f6d5c0f\") " Oct 31 04:09:44.995976 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:44.995951 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-openshift-ca" pod "7a456a46-63ad-4621-b430-a9980f6d5c0f" (UID: "7a456a46-63ad-4621-b430-a9980f6d5c0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:44.997612 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:44.997587 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-xks-ca" pod "7a456a46-63ad-4621-b430-a9980f6d5c0f" (UID: "7a456a46-63ad-4621-b430-a9980f6d5c0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:09:44.997684 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:44.997658 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a456a46-63ad-4621-b430-a9980f6d5c0f-kube-api-access-9tnf7" pod "7a456a46-63ad-4621-b430-a9980f6d5c0f" (UID: "7a456a46-63ad-4621-b430-a9980f6d5c0f"). InnerVolumeSpecName "kube-api-access-9tnf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:45.096864 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:45.096826 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:45.096864 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:45.096859 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tnf7\" (UniqueName: \"kubernetes.io/projected/7a456a46-63ad-4621-b430-a9980f6d5c0f-kube-api-access-9tnf7\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:45.096864 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:45.096872 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a456a46-63ad-4621-b430-a9980f6d5c0f-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:45.749857 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:45.749829 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" Oct 31 04:09:45.750061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:45.749863 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv" event={"ID":"7a456a46-63ad-4621-b430-a9980f6d5c0f","Type":"ContainerDied","Data":"89bc2cb272971291ce458ecf420fee691944238c47563ab2fcc743927f3113b7"} Oct 31 04:09:45.750061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:45.749926 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89bc2cb272971291ce458ecf420fee691944238c47563ab2fcc743927f3113b7" Oct 31 04:09:46.052319 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.052215 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r"] Oct 31 04:09:46.055711 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.055671 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec27414d-fk68r"] Oct 31 04:09:46.094194 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.094112 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6"] Oct 31 04:09:46.097934 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.097905 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf98297-xrdz6"] Oct 31 04:09:46.487341 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.487313 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:09:46.566889 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.566828 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:09:46.615967 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.615938 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-plugins-config-volume\") pod \"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e\" (UID: \"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e\") " Oct 31 04:09:46.616174 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.616026 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-kube-api-access-pfdtv\" (UniqueName: \"kubernetes.io/projected/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-kube-api-access-pfdtv\") pod \"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e\" (UID: \"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e\") " Oct 31 04:09:46.616370 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.616344 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-plugins-config-volume" pod "3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e" (UID: "3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:46.616450 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.616434 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:46.618136 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.618099 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-kube-api-access-pfdtv" pod "3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e" (UID: "3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e"). InnerVolumeSpecName "kube-api-access-pfdtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:46.765271 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.765234 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0182921c-a48d-47db-a101-5cd137abf755-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0182921c-a48d-47db-a101-5cd137abf755-plugins-config-volume\") pod \"0182921c-a48d-47db-a101-5cd137abf755\" (UID: \"0182921c-a48d-47db-a101-5cd137abf755\") " Oct 31 04:09:46.765473 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.765295 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0182921c-a48d-47db-a101-5cd137abf755-kube-api-access-v8cpp\" (UniqueName: \"kubernetes.io/projected/0182921c-a48d-47db-a101-5cd137abf755-kube-api-access-v8cpp\") pod \"0182921c-a48d-47db-a101-5cd137abf755\" (UID: \"0182921c-a48d-47db-a101-5cd137abf755\") " Oct 31 04:09:46.766021 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.765988 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfdtv\" (UniqueName: \"kubernetes.io/projected/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e-kube-api-access-pfdtv\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:46.767904 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.767869 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0182921c-a48d-47db-a101-5cd137abf755-plugins-config-volume" pod "0182921c-a48d-47db-a101-5cd137abf755" (UID: "0182921c-a48d-47db-a101-5cd137abf755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:46.768925 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.768897 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0182921c-a48d-47db-a101-5cd137abf755-kube-api-access-v8cpp" pod "0182921c-a48d-47db-a101-5cd137abf755" (UID: "0182921c-a48d-47db-a101-5cd137abf755"). InnerVolumeSpecName "kube-api-access-v8cpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:46.777616 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.777582 4426 generic.go:356] "Generic (PLEG): container finished" podID="0182921c-a48d-47db-a101-5cd137abf755" containerID="5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc" exitCode=0 Oct 31 04:09:46.777743 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.777704 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" Oct 31 04:09:46.777821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.777705 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" event={"ID":"0182921c-a48d-47db-a101-5cd137abf755","Type":"ContainerDied","Data":"5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc"} Oct 31 04:09:46.777879 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.777837 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v" event={"ID":"0182921c-a48d-47db-a101-5cd137abf755","Type":"ContainerDied","Data":"7e02ad3dd521915f0cd5c55bd8d8ab951b989cc6510a591c92acf1c53b30035b"} Oct 31 04:09:46.777879 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.777873 4426 scope.go:122] "RemoveContainer" containerID="5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc" Oct 31 04:09:46.779924 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.779896 4426 generic.go:356] "Generic (PLEG): container finished" podID="3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e" containerID="1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522" exitCode=0 Oct 31 04:09:46.780040 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.779976 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" Oct 31 04:09:46.780040 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.780000 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" event={"ID":"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e","Type":"ContainerDied","Data":"1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522"} Oct 31 04:09:46.780249 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.780068 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc" event={"ID":"3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e","Type":"ContainerDied","Data":"f53724ac96c308e23cd5fd5445cf7817a62bc3c78810284c713a3a650271773e"} Oct 31 04:09:46.792242 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.792220 4426 scope.go:122] "RemoveContainer" containerID="5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc" Oct 31 04:09:46.792550 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:09:46.792532 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc\": container with ID starting with 5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc not found: ID does not exist" containerID="5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc" Oct 31 04:09:46.792601 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.792558 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc"} err="failed to get container status \"5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc\": rpc error: code = NotFound desc = could not find container \"5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc\": container with ID starting with 5819f457f13464c7fbd7ce97ad66b2265a04410dfe2d9d6451e9f311eff7b6fc not found: ID does not exist" Oct 31 04:09:46.792601 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.792575 4426 scope.go:122] "RemoveContainer" containerID="1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522" Oct 31 04:09:46.806236 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.806212 4426 scope.go:122] "RemoveContainer" containerID="1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522" Oct 31 04:09:46.806573 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:09:46.806550 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522\": container with ID starting with 1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522 not found: ID does not exist" containerID="1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522" Oct 31 04:09:46.806638 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.806584 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522"} err="failed to get container status \"1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522\": rpc error: code = NotFound desc = could not find container \"1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522\": container with ID starting with 1df4e1019312f1b87eb5a30e1c4d29e1d1cad2e919498128d8ef11f1972d4522 not found: ID does not exist" Oct 31 04:09:46.866948 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.866862 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0182921c-a48d-47db-a101-5cd137abf755-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:46.866948 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.866893 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8cpp\" (UniqueName: \"kubernetes.io/projected/0182921c-a48d-47db-a101-5cd137abf755-kube-api-access-v8cpp\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:46.934627 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.934595 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc"] Oct 31 04:09:46.941042 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:46.941016 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-pgqbc"] Oct 31 04:09:47.040412 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:47.040376 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v"] Oct 31 04:09:47.045428 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:47.045392 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-fct2v"] Oct 31 04:09:47.129314 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:47.129240 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0182921c-a48d-47db-a101-5cd137abf755" path="/var/lib/kubelet/pods/0182921c-a48d-47db-a101-5cd137abf755/volumes" Oct 31 04:09:47.129687 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:47.129649 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e" path="/var/lib/kubelet/pods/3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e/volumes" Oct 31 04:09:47.130014 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:47.130001 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c91447-1f10-42d6-8702-377d3e5f1115" path="/var/lib/kubelet/pods/50c91447-1f10-42d6-8702-377d3e5f1115/volumes" Oct 31 04:09:47.130668 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:47.130654 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f947877-2492-415b-814b-fca6c1ee9bb8" path="/var/lib/kubelet/pods/5f947877-2492-415b-814b-fca6c1ee9bb8/volumes" Oct 31 04:09:47.859939 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:47.859904 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t"] Oct 31 04:09:47.863135 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:47.863100 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-288b63-n6q7t"] Oct 31 04:09:49.130572 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:49.130540 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb712b2-a972-4d26-966c-3ca3eb8b9da8" path="/var/lib/kubelet/pods/ebb712b2-a972-4d26-966c-3ca3eb8b9da8/volumes" Oct 31 04:09:51.003795 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.003760 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q"] Oct 31 04:09:51.004354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.004293 4426 cpu_manager.go:414] "containerMap: removing container" podUID="0182921c-a48d-47db-a101-5cd137abf755" containerName="payload-pre-processing" Oct 31 04:09:51.004354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.004308 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0182921c-a48d-47db-a101-5cd137abf755" containerName="payload-pre-processing" Oct 31 04:09:51.004354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.004320 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7a456a46-63ad-4621-b430-a9980f6d5c0f" containerName="revoke-keys" Oct 31 04:09:51.004354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.004325 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a456a46-63ad-4621-b430-a9980f6d5c0f" containerName="revoke-keys" Oct 31 04:09:51.004354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.004341 4426 cpu_manager.go:414] "containerMap: removing container" podUID="3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e" containerName="payload-processing" Oct 31 04:09:51.004354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.004346 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e" containerName="payload-processing" Oct 31 04:09:51.004698 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.004424 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="3583f9e6-d6fd-4d10-bcb8-a259c8fe9a6e" containerName="payload-processing" Oct 31 04:09:51.004698 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.004438 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="0182921c-a48d-47db-a101-5cd137abf755" containerName="payload-pre-processing" Oct 31 04:09:51.004698 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.004445 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a456a46-63ad-4621-b430-a9980f6d5c0f" containerName="revoke-keys" Oct 31 04:09:51.009691 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.009666 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:51.063867 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.063832 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q"] Oct 31 04:09:51.105665 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.105628 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:09:51.105880 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.105674 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:09:51.105880 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.105745 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:09:51.106069 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.106042 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:09:51.115433 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.115401 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:51.115568 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.115444 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cdf2\" (UniqueName: \"kubernetes.io/projected/ce585b22-bc58-4c49-84ae-0edad348350e-kube-api-access-7cdf2\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:51.115649 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.115627 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:51.216519 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.216478 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:51.216705 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.216680 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:51.216800 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.216706 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cdf2\" (UniqueName: \"kubernetes.io/projected/ce585b22-bc58-4c49-84ae-0edad348350e-kube-api-access-7cdf2\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:51.217482 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.217457 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:51.218871 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.218849 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:51.274386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.274300 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cdf2\" (UniqueName: \"kubernetes.io/projected/ce585b22-bc58-4c49-84ae-0edad348350e-kube-api-access-7cdf2\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:51.324170 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.324117 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:51.703623 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.703587 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q"] Oct 31 04:09:51.707833 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:09:51.707806 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce585b22_bc58_4c49_84ae_0edad348350e.slice/crio-f685ad226d66235fbaf6ed69d0df09322a7983bededdd32c9f7b0ec93af231fb WatchSource:0}: Error finding container f685ad226d66235fbaf6ed69d0df09322a7983bededdd32c9f7b0ec93af231fb: Status 404 returned error can't find the container with id f685ad226d66235fbaf6ed69d0df09322a7983bededdd32c9f7b0ec93af231fb Oct 31 04:09:51.820475 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:51.820438 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" event={"ID":"ce585b22-bc58-4c49-84ae-0edad348350e","Type":"ContainerStarted","Data":"f685ad226d66235fbaf6ed69d0df09322a7983bededdd32c9f7b0ec93af231fb"} Oct 31 04:09:52.830386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:52.830341 4426 generic.go:356] "Generic (PLEG): container finished" podID="ce585b22-bc58-4c49-84ae-0edad348350e" containerID="ed614e92698f8f4cf59fc33dbb558f02b5af2cd049a60bfcace25ca96792f12a" exitCode=0 Oct 31 04:09:52.830843 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:52.830416 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" event={"ID":"ce585b22-bc58-4c49-84ae-0edad348350e","Type":"ContainerDied","Data":"ed614e92698f8f4cf59fc33dbb558f02b5af2cd049a60bfcace25ca96792f12a"} Oct 31 04:09:53.995665 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:53.995636 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:54.045222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.045187 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-xks-ca\") pod \"ce585b22-bc58-4c49-84ae-0edad348350e\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " Oct 31 04:09:54.045422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.045257 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-openshift-ca\") pod \"ce585b22-bc58-4c49-84ae-0edad348350e\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " Oct 31 04:09:54.045422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.045314 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce585b22-bc58-4c49-84ae-0edad348350e-kube-api-access-7cdf2\" (UniqueName: \"kubernetes.io/projected/ce585b22-bc58-4c49-84ae-0edad348350e-kube-api-access-7cdf2\") pod \"ce585b22-bc58-4c49-84ae-0edad348350e\" (UID: \"ce585b22-bc58-4c49-84ae-0edad348350e\") " Oct 31 04:09:54.045692 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.045665 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-openshift-ca" pod "ce585b22-bc58-4c49-84ae-0edad348350e" (UID: "ce585b22-bc58-4c49-84ae-0edad348350e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:54.045787 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.045770 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:54.047455 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.047430 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce585b22-bc58-4c49-84ae-0edad348350e-kube-api-access-7cdf2" pod "ce585b22-bc58-4c49-84ae-0edad348350e" (UID: "ce585b22-bc58-4c49-84ae-0edad348350e"). InnerVolumeSpecName "kube-api-access-7cdf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:54.047545 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.047488 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-xks-ca" pod "ce585b22-bc58-4c49-84ae-0edad348350e" (UID: "ce585b22-bc58-4c49-84ae-0edad348350e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:09:54.146670 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.146636 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cdf2\" (UniqueName: \"kubernetes.io/projected/ce585b22-bc58-4c49-84ae-0edad348350e-kube-api-access-7cdf2\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:54.146670 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.146669 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce585b22-bc58-4c49-84ae-0edad348350e-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:54.856322 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.854397 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" event={"ID":"ce585b22-bc58-4c49-84ae-0edad348350e","Type":"ContainerDied","Data":"f685ad226d66235fbaf6ed69d0df09322a7983bededdd32c9f7b0ec93af231fb"} Oct 31 04:09:54.856322 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.854410 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q" Oct 31 04:09:54.856322 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:54.854461 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f685ad226d66235fbaf6ed69d0df09322a7983bededdd32c9f7b0ec93af231fb" Oct 31 04:09:55.911187 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.907323 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p"] Oct 31 04:09:55.911187 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.910085 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ce585b22-bc58-4c49-84ae-0edad348350e" containerName="revoke-keys" Oct 31 04:09:55.911187 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.910890 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce585b22-bc58-4c49-84ae-0edad348350e" containerName="revoke-keys" Oct 31 04:09:55.911187 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.911063 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce585b22-bc58-4c49-84ae-0edad348350e" containerName="revoke-keys" Oct 31 04:09:55.914909 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.914881 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:55.950928 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.950888 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p"] Oct 31 04:09:55.964276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.964248 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:09:55.977839 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.977789 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76qsg\" (UniqueName: \"kubernetes.io/projected/590bfd99-6af6-453e-9ef0-bd2f4197718c-kube-api-access-76qsg\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:55.978010 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.977847 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:55.978106 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.978076 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:55.990919 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.990890 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:09:55.991052 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.990948 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:09:55.991052 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:55.990993 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:09:56.079644 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:56.079604 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76qsg\" (UniqueName: \"kubernetes.io/projected/590bfd99-6af6-453e-9ef0-bd2f4197718c-kube-api-access-76qsg\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:56.079849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:56.079648 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:56.079849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:56.079786 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:56.080457 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:56.080434 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:56.082177 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:56.082154 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:56.137383 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:56.137349 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76qsg\" (UniqueName: \"kubernetes.io/projected/590bfd99-6af6-453e-9ef0-bd2f4197718c-kube-api-access-76qsg\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:56.231574 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:56.231488 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:56.609070 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:56.608994 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p"] Oct 31 04:09:56.613324 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:09:56.613294 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590bfd99_6af6_453e_9ef0_bd2f4197718c.slice/crio-a912113973fb963c05efb30817ddff09e4073c5ecbf7d3c7cbb769c848aa8d3c WatchSource:0}: Error finding container a912113973fb963c05efb30817ddff09e4073c5ecbf7d3c7cbb769c848aa8d3c: Status 404 returned error can't find the container with id a912113973fb963c05efb30817ddff09e4073c5ecbf7d3c7cbb769c848aa8d3c Oct 31 04:09:56.880591 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:56.880505 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" event={"ID":"590bfd99-6af6-453e-9ef0-bd2f4197718c","Type":"ContainerStarted","Data":"a912113973fb963c05efb30817ddff09e4073c5ecbf7d3c7cbb769c848aa8d3c"} Oct 31 04:09:57.889243 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:57.889161 4426 generic.go:356] "Generic (PLEG): container finished" podID="590bfd99-6af6-453e-9ef0-bd2f4197718c" containerID="c20eff528353cd6d7e15a4afb9d217ad5e055b75f890396c1230d98602c30442" exitCode=0 Oct 31 04:09:57.889621 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:57.889299 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" event={"ID":"590bfd99-6af6-453e-9ef0-bd2f4197718c","Type":"ContainerDied","Data":"c20eff528353cd6d7e15a4afb9d217ad5e055b75f890396c1230d98602c30442"} Oct 31 04:09:59.036292 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.036260 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:09:59.112314 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.112280 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-openshift-ca\") pod \"590bfd99-6af6-453e-9ef0-bd2f4197718c\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " Oct 31 04:09:59.112482 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.112345 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590bfd99-6af6-453e-9ef0-bd2f4197718c-kube-api-access-76qsg\" (UniqueName: \"kubernetes.io/projected/590bfd99-6af6-453e-9ef0-bd2f4197718c-kube-api-access-76qsg\") pod \"590bfd99-6af6-453e-9ef0-bd2f4197718c\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " Oct 31 04:09:59.112482 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.112450 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-xks-ca\") pod \"590bfd99-6af6-453e-9ef0-bd2f4197718c\" (UID: \"590bfd99-6af6-453e-9ef0-bd2f4197718c\") " Oct 31 04:09:59.112714 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.112683 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-openshift-ca" pod "590bfd99-6af6-453e-9ef0-bd2f4197718c" (UID: "590bfd99-6af6-453e-9ef0-bd2f4197718c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:59.112830 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.112810 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:59.114575 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.114550 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-xks-ca" pod "590bfd99-6af6-453e-9ef0-bd2f4197718c" (UID: "590bfd99-6af6-453e-9ef0-bd2f4197718c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:09:59.114575 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.114561 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590bfd99-6af6-453e-9ef0-bd2f4197718c-kube-api-access-76qsg" pod "590bfd99-6af6-453e-9ef0-bd2f4197718c" (UID: "590bfd99-6af6-453e-9ef0-bd2f4197718c"). InnerVolumeSpecName "kube-api-access-76qsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:59.214201 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.214089 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76qsg\" (UniqueName: \"kubernetes.io/projected/590bfd99-6af6-453e-9ef0-bd2f4197718c-kube-api-access-76qsg\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:59.214201 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.214153 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590bfd99-6af6-453e-9ef0-bd2f4197718c-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:59.370704 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.370665 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:09:59.921108 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.921063 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" event={"ID":"590bfd99-6af6-453e-9ef0-bd2f4197718c","Type":"ContainerDied","Data":"a912113973fb963c05efb30817ddff09e4073c5ecbf7d3c7cbb769c848aa8d3c"} Oct 31 04:09:59.921425 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.921400 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a912113973fb963c05efb30817ddff09e4073c5ecbf7d3c7cbb769c848aa8d3c" Oct 31 04:09:59.922440 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:09:59.922415 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p" Oct 31 04:10:02.848071 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:10:02.848031 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:10:07.340919 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:07.340882 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd"] Oct 31 04:10:12.342065 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:12.341993 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" podUID="e2466146-ded0-4cbe-98d2-168b8ad5227c" containerName="payload-processing" containerID="cri-o://1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764" gracePeriod=25 Oct 31 04:10:13.185689 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:13.185649 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" podUID="e2466146-ded0-4cbe-98d2-168b8ad5227c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:10:16.953059 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:16.952997 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q"] Oct 31 04:10:16.997306 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:16.997270 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-gkm4q"] Oct 31 04:10:17.129617 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:17.129579 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d2bc51-8084-4bb7-9366-801d380a6807" path="/var/lib/kubelet/pods/c4d2bc51-8084-4bb7-9366-801d380a6807/volumes" Oct 31 04:10:23.049309 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:23.049271 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4"] Oct 31 04:10:23.095761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:23.095729 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cf615e8-nmkf4"] Oct 31 04:10:23.129775 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:23.129739 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b304e4-6937-490d-82c1-24dafd31b2bc" path="/var/lib/kubelet/pods/92b304e4-6937-490d-82c1-24dafd31b2bc/volumes" Oct 31 04:10:23.185717 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:23.185676 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" podUID="e2466146-ded0-4cbe-98d2-168b8ad5227c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:10:26.094595 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.094557 4426 scope.go:122] "RemoveContainer" containerID="c3fc8d86446fa2112c3329517d7f02ce39d7d88dd8559d7c8ae3569c004ff302" Oct 31 04:10:26.105189 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.105165 4426 scope.go:122] "RemoveContainer" containerID="7e46679c5e9c8e5fdbb885952cc1dead2e226f7f7f0e13b19eaf9820ad7c0c0b" Oct 31 04:10:26.114940 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.114915 4426 scope.go:122] "RemoveContainer" containerID="9b84ec2fe583df23479694bf14e0fbf68198afa76dd9c663417af5d283f08929" Oct 31 04:10:26.125916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.125885 4426 scope.go:122] "RemoveContainer" containerID="fbe624bfce8764da44d394aa60a00b39646ff4b18f08505f70b2533eb6da1c34" Oct 31 04:10:26.613699 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.613662 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq"] Oct 31 04:10:26.614359 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.614335 4426 cpu_manager.go:414] "containerMap: removing container" podUID="590bfd99-6af6-453e-9ef0-bd2f4197718c" containerName="revoke-keys" Oct 31 04:10:26.614359 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.614359 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590bfd99-6af6-453e-9ef0-bd2f4197718c" containerName="revoke-keys" Oct 31 04:10:26.614571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.614526 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="590bfd99-6af6-453e-9ef0-bd2f4197718c" containerName="revoke-keys" Oct 31 04:10:26.620282 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.620257 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv"] Oct 31 04:10:26.620406 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.620375 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:26.625227 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.625202 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:26.669890 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.669856 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:10:26.669890 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.669891 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:10:26.733466 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.733426 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv"] Oct 31 04:10:26.752958 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.741378 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:10:26.752958 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.741528 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:10:26.752958 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.742093 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a458b0ed-dmgzv\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:26.752958 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.742171 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts28r\" (UniqueName: \"kubernetes.io/projected/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-kube-api-access-ts28r\") pod \"maas-api-revoke-sub-51e87883-gqztq\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:26.752958 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.742204 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a458b0ed-dmgzv\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:26.752958 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.742247 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e87883-gqztq\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:26.752958 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.742289 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e87883-gqztq\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:26.752958 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.742444 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdhfl\" (UniqueName: \"kubernetes.io/projected/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-kube-api-access-qdhfl\") pod \"maas-api-revoke-sub-a458b0ed-dmgzv\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:26.763224 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.763188 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq"] Oct 31 04:10:26.843910 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.843867 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e87883-gqztq\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:26.844157 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.843942 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e87883-gqztq\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:26.844157 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.844100 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdhfl\" (UniqueName: \"kubernetes.io/projected/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-kube-api-access-qdhfl\") pod \"maas-api-revoke-sub-a458b0ed-dmgzv\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:26.844157 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.844146 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a458b0ed-dmgzv\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:26.844419 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.844201 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts28r\" (UniqueName: \"kubernetes.io/projected/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-kube-api-access-ts28r\") pod \"maas-api-revoke-sub-51e87883-gqztq\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:26.844419 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.844227 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a458b0ed-dmgzv\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:26.844706 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.844679 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e87883-gqztq\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:26.845382 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.845354 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a458b0ed-dmgzv\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:26.846777 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.846753 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a458b0ed-dmgzv\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:26.847186 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.847169 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e87883-gqztq\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:26.906004 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.905919 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdhfl\" (UniqueName: \"kubernetes.io/projected/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-kube-api-access-qdhfl\") pod \"maas-api-revoke-sub-a458b0ed-dmgzv\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:26.908594 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.908368 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts28r\" (UniqueName: \"kubernetes.io/projected/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-kube-api-access-ts28r\") pod \"maas-api-revoke-sub-51e87883-gqztq\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:26.939796 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.939764 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:26.950146 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:26.950103 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:27.007836 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.007761 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw"] Oct 31 04:10:27.017328 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.015157 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:10:27.067164 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.065622 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6e81aea5-dockercfg-98j9b\"" Oct 31 04:10:27.067164 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.065621 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6e81aea5\"" Oct 31 04:10:27.067164 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.066049 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw"] Oct 31 04:10:27.169207 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.148730 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5kpk\" (UniqueName: \"kubernetes.io/projected/901174d2-5cca-4e9b-8d17-bfb128156b2c-kube-api-access-b5kpk\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw\" (UID: \"901174d2-5cca-4e9b-8d17-bfb128156b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:10:27.169207 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.148869 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901174d2-5cca-4e9b-8d17-bfb128156b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw\" (UID: \"901174d2-5cca-4e9b-8d17-bfb128156b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:10:27.207151 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.206933 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf"] Oct 31 04:10:27.212198 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.212163 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:10:27.234111 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.234081 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf"] Oct 31 04:10:27.250680 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.250643 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5kpk\" (UniqueName: \"kubernetes.io/projected/901174d2-5cca-4e9b-8d17-bfb128156b2c-kube-api-access-b5kpk\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw\" (UID: \"901174d2-5cca-4e9b-8d17-bfb128156b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:10:27.250843 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.250716 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gpfk\" (UniqueName: \"kubernetes.io/projected/e061f010-dc83-4cb6-ae96-ce0a382a935d-kube-api-access-7gpfk\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf\" (UID: \"e061f010-dc83-4cb6-ae96-ce0a382a935d\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:10:27.250843 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.250758 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901174d2-5cca-4e9b-8d17-bfb128156b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw\" (UID: \"901174d2-5cca-4e9b-8d17-bfb128156b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:10:27.250948 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.250899 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e061f010-dc83-4cb6-ae96-ce0a382a935d-plugins-config-volume\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf\" (UID: \"e061f010-dc83-4cb6-ae96-ce0a382a935d\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:10:27.251740 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.251713 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901174d2-5cca-4e9b-8d17-bfb128156b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw\" (UID: \"901174d2-5cca-4e9b-8d17-bfb128156b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:10:27.308203 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.308159 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5kpk\" (UniqueName: \"kubernetes.io/projected/901174d2-5cca-4e9b-8d17-bfb128156b2c-kube-api-access-b5kpk\") pod \"payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw\" (UID: \"901174d2-5cca-4e9b-8d17-bfb128156b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:10:27.329530 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.329499 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq"] Oct 31 04:10:27.330322 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:10:27.330298 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80afc3d_ea65_4fe4_a8cb_c8342be9f6e8.slice/crio-3c45bcbb13391817a91adaf4aa383d39b03bdba22d168930068b7533d82f7fda WatchSource:0}: Error finding container 3c45bcbb13391817a91adaf4aa383d39b03bdba22d168930068b7533d82f7fda: Status 404 returned error can't find the container with id 3c45bcbb13391817a91adaf4aa383d39b03bdba22d168930068b7533d82f7fda Oct 31 04:10:27.351693 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.351661 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gpfk\" (UniqueName: \"kubernetes.io/projected/e061f010-dc83-4cb6-ae96-ce0a382a935d-kube-api-access-7gpfk\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf\" (UID: \"e061f010-dc83-4cb6-ae96-ce0a382a935d\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:10:27.351812 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.351795 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e061f010-dc83-4cb6-ae96-ce0a382a935d-plugins-config-volume\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf\" (UID: \"e061f010-dc83-4cb6-ae96-ce0a382a935d\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:10:27.352517 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.352490 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e061f010-dc83-4cb6-ae96-ce0a382a935d-plugins-config-volume\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf\" (UID: \"e061f010-dc83-4cb6-ae96-ce0a382a935d\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:10:27.358286 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.358259 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:10:27.382343 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:10:27.382313 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7087dcc_6049_43ce_9bfa_bd19cec2b7bd.slice/crio-e30ede0890c9aeb1bcfb2f1c80bc38a1b5d9099b544e14625a5ed94866b15679 WatchSource:0}: Error finding container e30ede0890c9aeb1bcfb2f1c80bc38a1b5d9099b544e14625a5ed94866b15679: Status 404 returned error can't find the container with id e30ede0890c9aeb1bcfb2f1c80bc38a1b5d9099b544e14625a5ed94866b15679 Oct 31 04:10:27.385571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.385475 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv"] Oct 31 04:10:27.402165 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.402114 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:10:27.411583 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.411517 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gpfk\" (UniqueName: \"kubernetes.io/projected/e061f010-dc83-4cb6-ae96-ce0a382a935d-kube-api-access-7gpfk\") pod \"payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf\" (UID: \"e061f010-dc83-4cb6-ae96-ce0a382a935d\") " pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:10:27.529072 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.529038 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:10:27.732805 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.732775 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q"] Oct 31 04:10:27.736327 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.736297 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-938507-gbk9q"] Oct 31 04:10:27.759902 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.759865 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw"] Oct 31 04:10:27.765297 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:10:27.765267 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901174d2_5cca_4e9b_8d17_bfb128156b2c.slice/crio-947d8ede5bd92e87d28f741898fd6eab5a6baa9f060a58df36423caa3ee6c67b WatchSource:0}: Error finding container 947d8ede5bd92e87d28f741898fd6eab5a6baa9f060a58df36423caa3ee6c67b: Status 404 returned error can't find the container with id 947d8ede5bd92e87d28f741898fd6eab5a6baa9f060a58df36423caa3ee6c67b Oct 31 04:10:27.906493 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:27.906457 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf"] Oct 31 04:10:27.910367 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:10:27.910339 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode061f010_dc83_4cb6_ae96_ce0a382a935d.slice/crio-f09df8741e926e5a0e44a73c4210112ef4abf85bfeb4c14f6e4ac6bd5aa9c3ac WatchSource:0}: Error finding container f09df8741e926e5a0e44a73c4210112ef4abf85bfeb4c14f6e4ac6bd5aa9c3ac: Status 404 returned error can't find the container with id f09df8741e926e5a0e44a73c4210112ef4abf85bfeb4c14f6e4ac6bd5aa9c3ac Oct 31 04:10:28.210652 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.210529 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" event={"ID":"901174d2-5cca-4e9b-8d17-bfb128156b2c","Type":"ContainerStarted","Data":"be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0"} Oct 31 04:10:28.210652 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.210617 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" event={"ID":"901174d2-5cca-4e9b-8d17-bfb128156b2c","Type":"ContainerStarted","Data":"947d8ede5bd92e87d28f741898fd6eab5a6baa9f060a58df36423caa3ee6c67b"} Oct 31 04:10:28.211497 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.210766 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:10:28.212403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.212377 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" event={"ID":"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd","Type":"ContainerStarted","Data":"2b099c10c274a0ef724e8978e1d884cf4d73fdcd878b4b28997af9be275bc190"} Oct 31 04:10:28.212505 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.212430 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" event={"ID":"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd","Type":"ContainerStarted","Data":"e30ede0890c9aeb1bcfb2f1c80bc38a1b5d9099b544e14625a5ed94866b15679"} Oct 31 04:10:28.214053 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.214025 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" event={"ID":"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8","Type":"ContainerStarted","Data":"c9fc4774faf8b9009343eb9e6f84c9cfa72e0af421459b4c9eff01910dc381da"} Oct 31 04:10:28.214187 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.214074 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" event={"ID":"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8","Type":"ContainerStarted","Data":"3c45bcbb13391817a91adaf4aa383d39b03bdba22d168930068b7533d82f7fda"} Oct 31 04:10:28.216734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.216713 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" event={"ID":"e061f010-dc83-4cb6-ae96-ce0a382a935d","Type":"ContainerStarted","Data":"69a5545fcbb57850fd705706742f417253a85fe45c7de894b53c78b9d1dd2ded"} Oct 31 04:10:28.216834 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.216746 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" event={"ID":"e061f010-dc83-4cb6-ae96-ce0a382a935d","Type":"ContainerStarted","Data":"f09df8741e926e5a0e44a73c4210112ef4abf85bfeb4c14f6e4ac6bd5aa9c3ac"} Oct 31 04:10:28.216922 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.216874 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:10:28.323695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.323577 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" podStartSLOduration=2.3235612359999998 podStartE2EDuration="2.323561236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:28.319562105 +0000 UTC m=+2525.320187338" watchObservedRunningTime="2026-10-31 04:10:28.323561236 +0000 UTC m=+2525.324186468" Oct 31 04:10:28.430539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.430472 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" podStartSLOduration=2.43045101 podStartE2EDuration="2.43045101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:28.427518294 +0000 UTC m=+2525.428143523" watchObservedRunningTime="2026-10-31 04:10:28.43045101 +0000 UTC m=+2525.431076239" Oct 31 04:10:28.537038 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.536972 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" podStartSLOduration=2.536955843 podStartE2EDuration="2.536955843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:28.535257049 +0000 UTC m=+2525.535882315" watchObservedRunningTime="2026-10-31 04:10:28.536955843 +0000 UTC m=+2525.537581070" Oct 31 04:10:28.644507 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:28.644452 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" podStartSLOduration=1.644438668 podStartE2EDuration="1.644438668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:28.64287688 +0000 UTC m=+2525.643502107" watchObservedRunningTime="2026-10-31 04:10:28.644438668 +0000 UTC m=+2525.645063938" Oct 31 04:10:29.129515 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:29.129433 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce585b22-bc58-4c49-84ae-0edad348350e" path="/var/lib/kubelet/pods/ce585b22-bc58-4c49-84ae-0edad348350e/volumes" Oct 31 04:10:29.225621 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:29.225582 4426 generic.go:356] "Generic (PLEG): container finished" podID="d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8" containerID="c9fc4774faf8b9009343eb9e6f84c9cfa72e0af421459b4c9eff01910dc381da" exitCode=0 Oct 31 04:10:29.226085 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:29.225674 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" event={"ID":"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8","Type":"ContainerDied","Data":"c9fc4774faf8b9009343eb9e6f84c9cfa72e0af421459b4c9eff01910dc381da"} Oct 31 04:10:29.227449 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:29.227426 4426 generic.go:356] "Generic (PLEG): container finished" podID="c7087dcc-6049-43ce-9bfa-bd19cec2b7bd" containerID="2b099c10c274a0ef724e8978e1d884cf4d73fdcd878b4b28997af9be275bc190" exitCode=0 Oct 31 04:10:29.227564 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:29.227511 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" event={"ID":"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd","Type":"ContainerDied","Data":"2b099c10c274a0ef724e8978e1d884cf4d73fdcd878b4b28997af9be275bc190"} Oct 31 04:10:30.398337 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.398306 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:30.428587 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.428557 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:30.487564 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.487476 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-kube-api-access-ts28r\" (UniqueName: \"kubernetes.io/projected/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-kube-api-access-ts28r\") pod \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " Oct 31 04:10:30.487564 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.487552 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-openshift-ca\") pod \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " Oct 31 04:10:30.487806 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.487653 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-openshift-ca\") pod \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " Oct 31 04:10:30.487806 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.487759 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-xks-ca\") pod \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " Oct 31 04:10:30.487933 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.487820 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-xks-ca\") pod \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\" (UID: \"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8\") " Oct 31 04:10:30.487933 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.487851 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-kube-api-access-qdhfl\" (UniqueName: \"kubernetes.io/projected/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-kube-api-access-qdhfl\") pod \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\" (UID: \"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd\") " Oct 31 04:10:30.487933 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.487897 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-openshift-ca" pod "d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8" (UID: "d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:10:30.488099 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.488035 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-openshift-ca" pod "c7087dcc-6049-43ce-9bfa-bd19cec2b7bd" (UID: "c7087dcc-6049-43ce-9bfa-bd19cec2b7bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:10:30.488332 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.488298 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:30.488332 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.488335 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:30.489928 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.489897 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-xks-ca" pod "d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8" (UID: "d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:10:30.490045 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.489977 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-xks-ca" pod "c7087dcc-6049-43ce-9bfa-bd19cec2b7bd" (UID: "c7087dcc-6049-43ce-9bfa-bd19cec2b7bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:10:30.490045 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.489995 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-kube-api-access-ts28r" pod "d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8" (UID: "d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8"). InnerVolumeSpecName "kube-api-access-ts28r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:30.490178 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.490114 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-kube-api-access-qdhfl" pod "c7087dcc-6049-43ce-9bfa-bd19cec2b7bd" (UID: "c7087dcc-6049-43ce-9bfa-bd19cec2b7bd"). InnerVolumeSpecName "kube-api-access-qdhfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:30.588980 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.588940 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:30.588980 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.588975 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:30.588980 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.588986 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdhfl\" (UniqueName: \"kubernetes.io/projected/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd-kube-api-access-qdhfl\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:30.589251 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:30.588997 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts28r\" (UniqueName: \"kubernetes.io/projected/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8-kube-api-access-ts28r\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:31.244080 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:31.244048 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" Oct 31 04:10:31.244080 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:31.244072 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv" event={"ID":"c7087dcc-6049-43ce-9bfa-bd19cec2b7bd","Type":"ContainerDied","Data":"e30ede0890c9aeb1bcfb2f1c80bc38a1b5d9099b544e14625a5ed94866b15679"} Oct 31 04:10:31.244349 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:31.244154 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e30ede0890c9aeb1bcfb2f1c80bc38a1b5d9099b544e14625a5ed94866b15679" Oct 31 04:10:31.245590 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:31.245569 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" Oct 31 04:10:31.245721 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:31.245610 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq" event={"ID":"d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8","Type":"ContainerDied","Data":"3c45bcbb13391817a91adaf4aa383d39b03bdba22d168930068b7533d82f7fda"} Oct 31 04:10:31.245721 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:31.245655 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c45bcbb13391817a91adaf4aa383d39b03bdba22d168930068b7533d82f7fda" Oct 31 04:10:32.609422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:32.609353 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 04:10:32.713545 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:32.713494 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2466146-ded0-4cbe-98d2-168b8ad5227c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2466146-ded0-4cbe-98d2-168b8ad5227c-plugins-config-volume\") pod \"e2466146-ded0-4cbe-98d2-168b8ad5227c\" (UID: \"e2466146-ded0-4cbe-98d2-168b8ad5227c\") " Oct 31 04:10:32.713719 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:32.713572 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2466146-ded0-4cbe-98d2-168b8ad5227c-kube-api-access-x927c\" (UniqueName: \"kubernetes.io/projected/e2466146-ded0-4cbe-98d2-168b8ad5227c-kube-api-access-x927c\") pod \"e2466146-ded0-4cbe-98d2-168b8ad5227c\" (UID: \"e2466146-ded0-4cbe-98d2-168b8ad5227c\") " Oct 31 04:10:32.713993 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:32.713966 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2466146-ded0-4cbe-98d2-168b8ad5227c-plugins-config-volume" pod "e2466146-ded0-4cbe-98d2-168b8ad5227c" (UID: "e2466146-ded0-4cbe-98d2-168b8ad5227c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:10:32.715916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:32.715880 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2466146-ded0-4cbe-98d2-168b8ad5227c-kube-api-access-x927c" pod "e2466146-ded0-4cbe-98d2-168b8ad5227c" (UID: "e2466146-ded0-4cbe-98d2-168b8ad5227c"). InnerVolumeSpecName "kube-api-access-x927c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:32.814614 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:32.814577 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x927c\" (UniqueName: \"kubernetes.io/projected/e2466146-ded0-4cbe-98d2-168b8ad5227c-kube-api-access-x927c\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:32.814614 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:32.814612 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2466146-ded0-4cbe-98d2-168b8ad5227c-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:32.855420 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:10:32.855378 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:10:33.265225 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:33.265188 4426 generic.go:356] "Generic (PLEG): container finished" podID="e2466146-ded0-4cbe-98d2-168b8ad5227c" containerID="1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764" exitCode=0 Oct 31 04:10:33.265225 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:33.265226 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" event={"ID":"e2466146-ded0-4cbe-98d2-168b8ad5227c","Type":"ContainerDied","Data":"1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764"} Oct 31 04:10:33.265500 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:33.265249 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" Oct 31 04:10:33.265500 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:33.265268 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd" event={"ID":"e2466146-ded0-4cbe-98d2-168b8ad5227c","Type":"ContainerDied","Data":"1cde44f769397592e6798a9619723ff0bba3a6c694930830990e9a40983b7eac"} Oct 31 04:10:33.265500 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:33.265290 4426 scope.go:122] "RemoveContainer" containerID="1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764" Oct 31 04:10:33.275977 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:33.275954 4426 scope.go:122] "RemoveContainer" containerID="1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764" Oct 31 04:10:33.276287 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:10:33.276258 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764\": container with ID starting with 1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764 not found: ID does not exist" containerID="1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764" Oct 31 04:10:33.276383 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:33.276294 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764"} err="failed to get container status \"1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764\": rpc error: code = NotFound desc = could not find container \"1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764\": container with ID starting with 1a565d177e1f111bba89312b81ef0b273e207285e0d4deac569f063549f95764 not found: ID does not exist" Oct 31 04:10:33.420721 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:33.420685 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd"] Oct 31 04:10:33.467470 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:33.467431 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae8e27-5c56887bf5-9rqmd"] Oct 31 04:10:35.130525 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.130488 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2466146-ded0-4cbe-98d2-168b8ad5227c" path="/var/lib/kubelet/pods/e2466146-ded0-4cbe-98d2-168b8ad5227c/volumes" Oct 31 04:10:35.615851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.615814 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj"] Oct 31 04:10:35.616397 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.616380 4426 cpu_manager.go:414] "containerMap: removing container" podUID="e2466146-ded0-4cbe-98d2-168b8ad5227c" containerName="payload-processing" Oct 31 04:10:35.616397 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.616397 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2466146-ded0-4cbe-98d2-168b8ad5227c" containerName="payload-processing" Oct 31 04:10:35.616556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.616427 4426 cpu_manager.go:414] "containerMap: removing container" podUID="c7087dcc-6049-43ce-9bfa-bd19cec2b7bd" containerName="revoke-keys" Oct 31 04:10:35.616556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.616432 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7087dcc-6049-43ce-9bfa-bd19cec2b7bd" containerName="revoke-keys" Oct 31 04:10:35.616556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.616447 4426 cpu_manager.go:414] "containerMap: removing container" podUID="d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8" containerName="revoke-keys" Oct 31 04:10:35.616556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.616454 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8" containerName="revoke-keys" Oct 31 04:10:35.616556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.616526 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7087dcc-6049-43ce-9bfa-bd19cec2b7bd" containerName="revoke-keys" Oct 31 04:10:35.616556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.616538 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8" containerName="revoke-keys" Oct 31 04:10:35.616556 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.616549 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2466146-ded0-4cbe-98d2-168b8ad5227c" containerName="payload-processing" Oct 31 04:10:35.620802 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.620783 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:35.669703 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.669665 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:10:35.674197 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.674172 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj"] Oct 31 04:10:35.716367 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.716313 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:10:35.716555 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.716391 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:10:35.716555 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.716444 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:10:35.752951 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.752918 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fz49\" (UniqueName: \"kubernetes.io/projected/a7c952f9-409a-4199-b1e6-aaefddd99199-kube-api-access-7fz49\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:35.752951 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.752959 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:35.753220 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.752993 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:35.854512 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.854479 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fz49\" (UniqueName: \"kubernetes.io/projected/a7c952f9-409a-4199-b1e6-aaefddd99199-kube-api-access-7fz49\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:35.854732 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.854519 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:35.854732 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.854568 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:35.855364 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.855331 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:35.857119 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.857087 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:35.912256 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.912223 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fz49\" (UniqueName: \"kubernetes.io/projected/a7c952f9-409a-4199-b1e6-aaefddd99199-kube-api-access-7fz49\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:35.937966 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:35.937930 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:36.326202 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:36.326086 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj"] Oct 31 04:10:36.330611 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:10:36.330561 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c952f9_409a_4199_b1e6_aaefddd99199.slice/crio-930f10504ecabec4e802e8c0f67af500bc449c734b11243d1fb99f053df9ef4c WatchSource:0}: Error finding container 930f10504ecabec4e802e8c0f67af500bc449c734b11243d1fb99f053df9ef4c: Status 404 returned error can't find the container with id 930f10504ecabec4e802e8c0f67af500bc449c734b11243d1fb99f053df9ef4c Oct 31 04:10:37.298351 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:37.298309 4426 generic.go:356] "Generic (PLEG): container finished" podID="a7c952f9-409a-4199-b1e6-aaefddd99199" containerID="ca0611c7d8d5579f384c8b2b9bad1c2c94f26685e46b8ddf1c1c6605349105e9" exitCode=0 Oct 31 04:10:37.298545 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:37.298391 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" event={"ID":"a7c952f9-409a-4199-b1e6-aaefddd99199","Type":"ContainerDied","Data":"ca0611c7d8d5579f384c8b2b9bad1c2c94f26685e46b8ddf1c1c6605349105e9"} Oct 31 04:10:37.298545 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:37.298443 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" event={"ID":"a7c952f9-409a-4199-b1e6-aaefddd99199","Type":"ContainerStarted","Data":"930f10504ecabec4e802e8c0f67af500bc449c734b11243d1fb99f053df9ef4c"} Oct 31 04:10:38.502242 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:38.502211 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:38.586014 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:38.585913 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-xks-ca\") pod \"a7c952f9-409a-4199-b1e6-aaefddd99199\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " Oct 31 04:10:38.586014 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:38.585977 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-openshift-ca\") pod \"a7c952f9-409a-4199-b1e6-aaefddd99199\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " Oct 31 04:10:38.586278 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:38.586215 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c952f9-409a-4199-b1e6-aaefddd99199-kube-api-access-7fz49\" (UniqueName: \"kubernetes.io/projected/a7c952f9-409a-4199-b1e6-aaefddd99199-kube-api-access-7fz49\") pod \"a7c952f9-409a-4199-b1e6-aaefddd99199\" (UID: \"a7c952f9-409a-4199-b1e6-aaefddd99199\") " Oct 31 04:10:38.586863 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:38.586832 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-openshift-ca" pod "a7c952f9-409a-4199-b1e6-aaefddd99199" (UID: "a7c952f9-409a-4199-b1e6-aaefddd99199"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:10:38.588802 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:38.588772 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-xks-ca" pod "a7c952f9-409a-4199-b1e6-aaefddd99199" (UID: "a7c952f9-409a-4199-b1e6-aaefddd99199"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:10:38.589449 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:38.589420 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c952f9-409a-4199-b1e6-aaefddd99199-kube-api-access-7fz49" pod "a7c952f9-409a-4199-b1e6-aaefddd99199" (UID: "a7c952f9-409a-4199-b1e6-aaefddd99199"). InnerVolumeSpecName "kube-api-access-7fz49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:38.688043 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:38.688008 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fz49\" (UniqueName: \"kubernetes.io/projected/a7c952f9-409a-4199-b1e6-aaefddd99199-kube-api-access-7fz49\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:38.688043 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:38.688039 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:38.688043 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:38.688050 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c952f9-409a-4199-b1e6-aaefddd99199-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:39.230970 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:39.230941 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:10:39.231196 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:39.231013 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:10:39.322774 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:39.322746 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" Oct 31 04:10:39.322945 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:39.322777 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj" event={"ID":"a7c952f9-409a-4199-b1e6-aaefddd99199","Type":"ContainerDied","Data":"930f10504ecabec4e802e8c0f67af500bc449c734b11243d1fb99f053df9ef4c"} Oct 31 04:10:39.322945 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:39.322839 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="930f10504ecabec4e802e8c0f67af500bc449c734b11243d1fb99f053df9ef4c" Oct 31 04:10:47.057540 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:47.057501 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js"] Oct 31 04:10:47.160566 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:47.160535 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm"] Oct 31 04:10:50.900344 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:50.900300 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:10:52.057966 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.057911 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" podUID="dfb84fa0-c690-40cd-b19e-db6b1767bc34" containerName="payload-processing" containerID="cri-o://c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b" gracePeriod=25 Oct 31 04:10:52.077790 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:10:52.077756 4426 logging.go:55] [core] [Channel #6145 SubChannel #6146]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.180:50052", ServerName: "10.133.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.180:50052: connect: connection refused" Oct 31 04:10:52.161262 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.161190 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" podUID="994007af-8bf3-414d-9492-7934ccdbcd22" containerName="payload-pre-processing" containerID="cri-o://2a1c5d981309d74e16db3f2b0b6fce2e1758facc4021a5a5598d44736f2a6141" gracePeriod=25 Oct 31 04:10:52.257168 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.256585 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p"] Oct 31 04:10:52.295468 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.294820 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae8e27-qb26p"] Oct 31 04:10:52.429175 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.428188 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 04:10:52.438523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.438484 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb84fa0-c690-40cd-b19e-db6b1767bc34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb84fa0-c690-40cd-b19e-db6b1767bc34-plugins-config-volume\") pod \"dfb84fa0-c690-40cd-b19e-db6b1767bc34\" (UID: \"dfb84fa0-c690-40cd-b19e-db6b1767bc34\") " Oct 31 04:10:52.438523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.438524 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb84fa0-c690-40cd-b19e-db6b1767bc34-kube-api-access-bx5nw\" (UniqueName: \"kubernetes.io/projected/dfb84fa0-c690-40cd-b19e-db6b1767bc34-kube-api-access-bx5nw\") pod \"dfb84fa0-c690-40cd-b19e-db6b1767bc34\" (UID: \"dfb84fa0-c690-40cd-b19e-db6b1767bc34\") " Oct 31 04:10:52.438843 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.438621 4426 generic.go:356] "Generic (PLEG): container finished" podID="994007af-8bf3-414d-9492-7934ccdbcd22" containerID="2a1c5d981309d74e16db3f2b0b6fce2e1758facc4021a5a5598d44736f2a6141" exitCode=0 Oct 31 04:10:52.438843 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.438694 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" event={"ID":"994007af-8bf3-414d-9492-7934ccdbcd22","Type":"ContainerDied","Data":"2a1c5d981309d74e16db3f2b0b6fce2e1758facc4021a5a5598d44736f2a6141"} Oct 31 04:10:52.439119 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.439091 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb84fa0-c690-40cd-b19e-db6b1767bc34-plugins-config-volume" pod "dfb84fa0-c690-40cd-b19e-db6b1767bc34" (UID: "dfb84fa0-c690-40cd-b19e-db6b1767bc34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:10:52.446754 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.441068 4426 generic.go:356] "Generic (PLEG): container finished" podID="dfb84fa0-c690-40cd-b19e-db6b1767bc34" containerID="c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b" exitCode=0 Oct 31 04:10:52.446754 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.441151 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb84fa0-c690-40cd-b19e-db6b1767bc34-kube-api-access-bx5nw" pod "dfb84fa0-c690-40cd-b19e-db6b1767bc34" (UID: "dfb84fa0-c690-40cd-b19e-db6b1767bc34"). InnerVolumeSpecName "kube-api-access-bx5nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:52.446754 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.441262 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" Oct 31 04:10:52.446754 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.441365 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" event={"ID":"dfb84fa0-c690-40cd-b19e-db6b1767bc34","Type":"ContainerDied","Data":"c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b"} Oct 31 04:10:52.446754 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.441415 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" event={"ID":"dfb84fa0-c690-40cd-b19e-db6b1767bc34","Type":"ContainerDied","Data":"92d4671acbcf81cfcb6c1806a0badb660251a81bccd043314e4d36fd39cbc271"} Oct 31 04:10:52.446754 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.441444 4426 scope.go:122] "RemoveContainer" containerID="c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b" Oct 31 04:10:52.457732 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.457709 4426 scope.go:122] "RemoveContainer" containerID="c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b" Oct 31 04:10:52.458053 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:10:52.458032 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b\": container with ID starting with c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b not found: ID does not exist" containerID="c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b" Oct 31 04:10:52.458168 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.458067 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b"} err="failed to get container status \"c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b\": rpc error: code = NotFound desc = could not find container \"c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b\": container with ID starting with c3b92fc77079c89d64422641b4d149a501383420f05932bacfdd44ec4cedb01b not found: ID does not exist" Oct 31 04:10:52.489580 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.489549 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 04:10:52.540177 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.539858 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994007af-8bf3-414d-9492-7934ccdbcd22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994007af-8bf3-414d-9492-7934ccdbcd22-plugins-config-volume\") pod \"994007af-8bf3-414d-9492-7934ccdbcd22\" (UID: \"994007af-8bf3-414d-9492-7934ccdbcd22\") " Oct 31 04:10:52.540177 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.539904 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994007af-8bf3-414d-9492-7934ccdbcd22-kube-api-access-gjbn6\" (UniqueName: \"kubernetes.io/projected/994007af-8bf3-414d-9492-7934ccdbcd22-kube-api-access-gjbn6\") pod \"994007af-8bf3-414d-9492-7934ccdbcd22\" (UID: \"994007af-8bf3-414d-9492-7934ccdbcd22\") " Oct 31 04:10:52.540408 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.540273 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb84fa0-c690-40cd-b19e-db6b1767bc34-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.540408 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.540298 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx5nw\" (UniqueName: \"kubernetes.io/projected/dfb84fa0-c690-40cd-b19e-db6b1767bc34-kube-api-access-bx5nw\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.541023 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.540976 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994007af-8bf3-414d-9492-7934ccdbcd22-plugins-config-volume" pod "994007af-8bf3-414d-9492-7934ccdbcd22" (UID: "994007af-8bf3-414d-9492-7934ccdbcd22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:10:52.543067 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.543036 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994007af-8bf3-414d-9492-7934ccdbcd22-kube-api-access-gjbn6" pod "994007af-8bf3-414d-9492-7934ccdbcd22" (UID: "994007af-8bf3-414d-9492-7934ccdbcd22"). InnerVolumeSpecName "kube-api-access-gjbn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:52.641222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.641189 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994007af-8bf3-414d-9492-7934ccdbcd22-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.641368 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.641224 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjbn6\" (UniqueName: \"kubernetes.io/projected/994007af-8bf3-414d-9492-7934ccdbcd22-kube-api-access-gjbn6\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.754475 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.754428 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js"] Oct 31 04:10:52.757988 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:52.757956 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js"] Oct 31 04:10:53.078341 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:53.078306 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-440092-5d59845559-827js" podUID="dfb84fa0-c690-40cd-b19e-db6b1767bc34" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.180:50052\" within 1s: context deadline exceeded" Oct 31 04:10:53.078711 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:10:53.078403 4426 logging.go:55] [core] [Channel #6145 SubChannel #6146]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.180:50052", ServerName: "10.133.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.180:50052: operation was canceled" Oct 31 04:10:53.130991 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:53.130960 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590bfd99-6af6-453e-9ef0-bd2f4197718c" path="/var/lib/kubelet/pods/590bfd99-6af6-453e-9ef0-bd2f4197718c/volumes" Oct 31 04:10:53.131470 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:53.131451 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb84fa0-c690-40cd-b19e-db6b1767bc34" path="/var/lib/kubelet/pods/dfb84fa0-c690-40cd-b19e-db6b1767bc34/volumes" Oct 31 04:10:53.450283 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:53.450249 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" Oct 31 04:10:53.450450 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:53.450248 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm" event={"ID":"994007af-8bf3-414d-9492-7934ccdbcd22","Type":"ContainerDied","Data":"3509b076b31c2e7a041c5a5a8b7331cc249ab1178622736196a521faac186172"} Oct 31 04:10:53.450450 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:53.450386 4426 scope.go:122] "RemoveContainer" containerID="2a1c5d981309d74e16db3f2b0b6fce2e1758facc4021a5a5598d44736f2a6141" Oct 31 04:10:53.549642 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:53.549611 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj"] Oct 31 04:10:53.553315 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:53.553290 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-440092-hpnsj"] Oct 31 04:10:53.655433 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:53.655400 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm"] Oct 31 04:10:53.683647 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:53.683616 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-440092-7ccf78f69f-s9srm"] Oct 31 04:10:55.129497 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:55.129455 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994007af-8bf3-414d-9492-7934ccdbcd22" path="/var/lib/kubelet/pods/994007af-8bf3-414d-9492-7934ccdbcd22/volumes" Oct 31 04:10:55.130108 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:10:55.130087 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c952f9-409a-4199-b1e6-aaefddd99199" path="/var/lib/kubelet/pods/a7c952f9-409a-4199-b1e6-aaefddd99199/volumes" Oct 31 04:11:02.846647 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:11:02.846605 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:11:11.845163 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.845048 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277"] Oct 31 04:11:11.845796 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.845777 4426 cpu_manager.go:414] "containerMap: removing container" podUID="a7c952f9-409a-4199-b1e6-aaefddd99199" containerName="revoke-keys" Oct 31 04:11:11.845845 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.845797 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c952f9-409a-4199-b1e6-aaefddd99199" containerName="revoke-keys" Oct 31 04:11:11.845845 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.845820 4426 cpu_manager.go:414] "containerMap: removing container" podUID="dfb84fa0-c690-40cd-b19e-db6b1767bc34" containerName="payload-processing" Oct 31 04:11:11.845845 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.845825 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb84fa0-c690-40cd-b19e-db6b1767bc34" containerName="payload-processing" Oct 31 04:11:11.845945 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.845850 4426 cpu_manager.go:414] "containerMap: removing container" podUID="994007af-8bf3-414d-9492-7934ccdbcd22" containerName="payload-pre-processing" Oct 31 04:11:11.845945 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.845857 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994007af-8bf3-414d-9492-7934ccdbcd22" containerName="payload-pre-processing" Oct 31 04:11:11.845945 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.845928 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c952f9-409a-4199-b1e6-aaefddd99199" containerName="revoke-keys" Oct 31 04:11:11.845945 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.845939 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb84fa0-c690-40cd-b19e-db6b1767bc34" containerName="payload-processing" Oct 31 04:11:11.846075 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.845949 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="994007af-8bf3-414d-9492-7934ccdbcd22" containerName="payload-pre-processing" Oct 31 04:11:11.850911 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.850890 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:11.946846 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.946810 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:11:11.947084 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.947002 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:11:11.947084 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.947010 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:11:11.947084 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.947062 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:11:11.950832 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.950804 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277"] Oct 31 04:11:11.970519 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.970486 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e7052-450d-477c-9424-205a73356be7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:11.970717 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.970688 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e7052-450d-477c-9424-205a73356be7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:11.970785 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:11.970739 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmz7\" (UniqueName: \"kubernetes.io/projected/497e7052-450d-477c-9424-205a73356be7-kube-api-access-rtmz7\") pod \"maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:12.073346 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:12.072110 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e7052-450d-477c-9424-205a73356be7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:12.073346 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:12.072435 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmz7\" (UniqueName: \"kubernetes.io/projected/497e7052-450d-477c-9424-205a73356be7-kube-api-access-rtmz7\") pod \"maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:12.073346 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:12.072572 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e7052-450d-477c-9424-205a73356be7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:12.073346 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:12.072968 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e7052-450d-477c-9424-205a73356be7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:12.092005 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:12.091894 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e7052-450d-477c-9424-205a73356be7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:12.130808 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:12.130721 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmz7\" (UniqueName: \"kubernetes.io/projected/497e7052-450d-477c-9424-205a73356be7-kube-api-access-rtmz7\") pod \"maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:12.164914 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:12.164878 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:12.542383 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:12.542345 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277"] Oct 31 04:11:12.546940 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:11:12.546903 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497e7052_450d_477c_9424_205a73356be7.slice/crio-5c4c49beb644e9795bf1235a0d826887b3fee0de72fce534248b836c675fd5eb WatchSource:0}: Error finding container 5c4c49beb644e9795bf1235a0d826887b3fee0de72fce534248b836c675fd5eb: Status 404 returned error can't find the container with id 5c4c49beb644e9795bf1235a0d826887b3fee0de72fce534248b836c675fd5eb Oct 31 04:11:12.598567 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:12.598515 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" event={"ID":"497e7052-450d-477c-9424-205a73356be7","Type":"ContainerStarted","Data":"5c4c49beb644e9795bf1235a0d826887b3fee0de72fce534248b836c675fd5eb"} Oct 31 04:11:13.616278 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:13.616193 4426 generic.go:356] "Generic (PLEG): container finished" podID="497e7052-450d-477c-9424-205a73356be7" containerID="1845b5d24bcff5859420fdb5694bc9c5579ce00ab44119ea03bd2cf56c3b46ab" exitCode=0 Oct 31 04:11:13.616780 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:13.616377 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" event={"ID":"497e7052-450d-477c-9424-205a73356be7","Type":"ContainerDied","Data":"1845b5d24bcff5859420fdb5694bc9c5579ce00ab44119ea03bd2cf56c3b46ab"} Oct 31 04:11:14.783955 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:14.783926 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:14.904156 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:14.904109 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497e7052-450d-477c-9424-205a73356be7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e7052-450d-477c-9424-205a73356be7-maas-api-xks-ca\") pod \"497e7052-450d-477c-9424-205a73356be7\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " Oct 31 04:11:14.904382 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:14.904214 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497e7052-450d-477c-9424-205a73356be7-kube-api-access-rtmz7\" (UniqueName: \"kubernetes.io/projected/497e7052-450d-477c-9424-205a73356be7-kube-api-access-rtmz7\") pod \"497e7052-450d-477c-9424-205a73356be7\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " Oct 31 04:11:14.904382 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:14.904267 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497e7052-450d-477c-9424-205a73356be7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e7052-450d-477c-9424-205a73356be7-maas-api-openshift-ca\") pod \"497e7052-450d-477c-9424-205a73356be7\" (UID: \"497e7052-450d-477c-9424-205a73356be7\") " Oct 31 04:11:14.904880 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:14.904846 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497e7052-450d-477c-9424-205a73356be7-maas-api-openshift-ca" pod "497e7052-450d-477c-9424-205a73356be7" (UID: "497e7052-450d-477c-9424-205a73356be7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:14.906901 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:14.906864 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497e7052-450d-477c-9424-205a73356be7-kube-api-access-rtmz7" pod "497e7052-450d-477c-9424-205a73356be7" (UID: "497e7052-450d-477c-9424-205a73356be7"). InnerVolumeSpecName "kube-api-access-rtmz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:14.910253 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:14.910225 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497e7052-450d-477c-9424-205a73356be7-maas-api-xks-ca" pod "497e7052-450d-477c-9424-205a73356be7" (UID: "497e7052-450d-477c-9424-205a73356be7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:15.005698 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.005667 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e7052-450d-477c-9424-205a73356be7-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:15.005698 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.005697 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e7052-450d-477c-9424-205a73356be7-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:15.005908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.005707 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmz7\" (UniqueName: \"kubernetes.io/projected/497e7052-450d-477c-9424-205a73356be7-kube-api-access-rtmz7\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:15.097571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.097533 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm"] Oct 31 04:11:15.098419 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.098398 4426 cpu_manager.go:414] "containerMap: removing container" podUID="497e7052-450d-477c-9424-205a73356be7" containerName="revoke-keys" Oct 31 04:11:15.098472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.098423 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497e7052-450d-477c-9424-205a73356be7" containerName="revoke-keys" Oct 31 04:11:15.098571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.098559 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="497e7052-450d-477c-9424-205a73356be7" containerName="revoke-keys" Oct 31 04:11:15.102217 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.102193 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:15.156452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.156361 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm"] Oct 31 04:11:15.208673 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.208633 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:15.208673 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.208672 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:15.208897 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.208706 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p65kd\" (UniqueName: \"kubernetes.io/projected/d0e63911-ac83-45dc-bb91-64de3d67d4a5-kube-api-access-p65kd\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:15.310450 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.310412 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:15.310450 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.310449 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:15.310721 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.310500 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65kd\" (UniqueName: \"kubernetes.io/projected/d0e63911-ac83-45dc-bb91-64de3d67d4a5-kube-api-access-p65kd\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:15.311330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.311298 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:15.313615 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.313589 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:15.368726 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.368691 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65kd\" (UniqueName: \"kubernetes.io/projected/d0e63911-ac83-45dc-bb91-64de3d67d4a5-kube-api-access-p65kd\") pod \"maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:15.418136 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.418022 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:15.636441 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.636413 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" Oct 31 04:11:15.636603 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.636414 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277" event={"ID":"497e7052-450d-477c-9424-205a73356be7","Type":"ContainerDied","Data":"5c4c49beb644e9795bf1235a0d826887b3fee0de72fce534248b836c675fd5eb"} Oct 31 04:11:15.636603 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.636550 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4c49beb644e9795bf1235a0d826887b3fee0de72fce534248b836c675fd5eb" Oct 31 04:11:15.803648 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:15.803562 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm"] Oct 31 04:11:15.808564 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:11:15.808538 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e63911_ac83_45dc_bb91_64de3d67d4a5.slice/crio-ac722e05432f0dcbf53f4f57d9e08f4420079323988ff8af25dccdbc730b0fc0 WatchSource:0}: Error finding container ac722e05432f0dcbf53f4f57d9e08f4420079323988ff8af25dccdbc730b0fc0: Status 404 returned error can't find the container with id ac722e05432f0dcbf53f4f57d9e08f4420079323988ff8af25dccdbc730b0fc0 Oct 31 04:11:16.367725 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:16.367687 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d"] Oct 31 04:11:16.646158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:16.646105 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" event={"ID":"d0e63911-ac83-45dc-bb91-64de3d67d4a5","Type":"ContainerStarted","Data":"323290b7ea403cf2e4ce5343eae5c0c546cbf27ba2c263d91c8649b5cb702e09"} Oct 31 04:11:16.646371 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:16.646182 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" event={"ID":"d0e63911-ac83-45dc-bb91-64de3d67d4a5","Type":"ContainerStarted","Data":"ac722e05432f0dcbf53f4f57d9e08f4420079323988ff8af25dccdbc730b0fc0"} Oct 31 04:11:16.756646 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:16.756587 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" podStartSLOduration=1.756570873 podStartE2EDuration="1.756570873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:16.755483538 +0000 UTC m=+2573.756108767" watchObservedRunningTime="2026-10-31 04:11:16.756570873 +0000 UTC m=+2573.757196100" Oct 31 04:11:16.903239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:16.903147 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:11:17.659917 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:17.659872 4426 generic.go:356] "Generic (PLEG): container finished" podID="d0e63911-ac83-45dc-bb91-64de3d67d4a5" containerID="323290b7ea403cf2e4ce5343eae5c0c546cbf27ba2c263d91c8649b5cb702e09" exitCode=0 Oct 31 04:11:17.660148 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:17.659938 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" event={"ID":"d0e63911-ac83-45dc-bb91-64de3d67d4a5","Type":"ContainerDied","Data":"323290b7ea403cf2e4ce5343eae5c0c546cbf27ba2c263d91c8649b5cb702e09"} Oct 31 04:11:18.806331 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:18.806304 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:18.848443 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:18.848407 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-xks-ca\") pod \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " Oct 31 04:11:18.848621 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:18.848459 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-openshift-ca\") pod \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " Oct 31 04:11:18.848621 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:18.848506 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e63911-ac83-45dc-bb91-64de3d67d4a5-kube-api-access-p65kd\" (UniqueName: \"kubernetes.io/projected/d0e63911-ac83-45dc-bb91-64de3d67d4a5-kube-api-access-p65kd\") pod \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\" (UID: \"d0e63911-ac83-45dc-bb91-64de3d67d4a5\") " Oct 31 04:11:18.848888 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:18.848857 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-openshift-ca" pod "d0e63911-ac83-45dc-bb91-64de3d67d4a5" (UID: "d0e63911-ac83-45dc-bb91-64de3d67d4a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:18.849048 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:18.849026 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:18.851169 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:18.851143 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e63911-ac83-45dc-bb91-64de3d67d4a5-kube-api-access-p65kd" pod "d0e63911-ac83-45dc-bb91-64de3d67d4a5" (UID: "d0e63911-ac83-45dc-bb91-64de3d67d4a5"). InnerVolumeSpecName "kube-api-access-p65kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:18.851169 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:18.851154 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-xks-ca" pod "d0e63911-ac83-45dc-bb91-64de3d67d4a5" (UID: "d0e63911-ac83-45dc-bb91-64de3d67d4a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:18.950346 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:18.950259 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e63911-ac83-45dc-bb91-64de3d67d4a5-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:18.950346 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:18.950287 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p65kd\" (UniqueName: \"kubernetes.io/projected/d0e63911-ac83-45dc-bb91-64de3d67d4a5-kube-api-access-p65kd\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:19.678531 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:19.678498 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" Oct 31 04:11:19.678531 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:19.678526 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm" event={"ID":"d0e63911-ac83-45dc-bb91-64de3d67d4a5","Type":"ContainerDied","Data":"ac722e05432f0dcbf53f4f57d9e08f4420079323988ff8af25dccdbc730b0fc0"} Oct 31 04:11:19.678774 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:19.678583 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac722e05432f0dcbf53f4f57d9e08f4420079323988ff8af25dccdbc730b0fc0" Oct 31 04:11:23.396944 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:23.396907 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8"] Oct 31 04:11:25.605086 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:25.605050 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf"] Oct 31 04:11:25.704812 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:25.704774 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw"] Oct 31 04:11:25.963914 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:25.963881 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr"] Oct 31 04:11:25.964434 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:25.964418 4426 cpu_manager.go:414] "containerMap: removing container" podUID="d0e63911-ac83-45dc-bb91-64de3d67d4a5" containerName="revoke-keys" Oct 31 04:11:25.964434 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:25.964434 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e63911-ac83-45dc-bb91-64de3d67d4a5" containerName="revoke-keys" Oct 31 04:11:25.964555 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:25.964520 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e63911-ac83-45dc-bb91-64de3d67d4a5" containerName="revoke-keys" Oct 31 04:11:25.967947 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:25.967930 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:26.054668 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.054629 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:11:26.054996 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.054978 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:11:26.055612 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.055589 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:11:26.055815 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.055800 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:11:26.057202 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.056572 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdv2p\" (UniqueName: \"kubernetes.io/projected/58c85669-cd8d-4766-b519-e2294dc5fc20-kube-api-access-vdv2p\") pod \"maas-api-revoke-sub-e12a23a2-bkqxr\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:26.057202 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.056781 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e12a23a2-bkqxr\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:26.057202 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.056836 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e12a23a2-bkqxr\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:26.057202 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.056941 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv"] Oct 31 04:11:26.067884 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.067862 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr"] Oct 31 04:11:26.068029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.068005 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:26.076227 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.076193 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv"] Oct 31 04:11:26.158971 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.158932 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862a6eb4-x6zjv\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:26.159194 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.158991 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e12a23a2-bkqxr\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:26.159194 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.159095 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e12a23a2-bkqxr\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:26.159290 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.159220 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqlhz\" (UniqueName: \"kubernetes.io/projected/ca8d3fac-50c1-4d10-9138-35446c73f10c-kube-api-access-zqlhz\") pod \"maas-api-revoke-sub-862a6eb4-x6zjv\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:26.159290 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.159260 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862a6eb4-x6zjv\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:26.159412 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.159385 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdv2p\" (UniqueName: \"kubernetes.io/projected/58c85669-cd8d-4766-b519-e2294dc5fc20-kube-api-access-vdv2p\") pod \"maas-api-revoke-sub-e12a23a2-bkqxr\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:26.159836 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.159816 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e12a23a2-bkqxr\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:26.161582 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.161563 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e12a23a2-bkqxr\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:26.198821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.198791 4426 scope.go:122] "RemoveContainer" containerID="fc05b1d6b9fa56b0c303274db1b8da7963ba3e48876f79745aa86c8539dc71ff" Oct 31 04:11:26.210364 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.210341 4426 scope.go:122] "RemoveContainer" containerID="c4859941939aaf5b08782e192a8d47055aea494807b066e6447f6331ff468d1c" Oct 31 04:11:26.217325 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.217254 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdv2p\" (UniqueName: \"kubernetes.io/projected/58c85669-cd8d-4766-b519-e2294dc5fc20-kube-api-access-vdv2p\") pod \"maas-api-revoke-sub-e12a23a2-bkqxr\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:26.222448 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.222374 4426 scope.go:122] "RemoveContainer" containerID="f4c59251e8397167fe6a075a3d41254a231cfb834242ba3963dff6ae91f9b4e8" Oct 31 04:11:26.234736 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.234712 4426 scope.go:122] "RemoveContainer" containerID="3db59308214ae53fe95cba62b04c643fb67888a3a1603a1439e092a7a1c8308d" Oct 31 04:11:26.246691 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.246669 4426 scope.go:122] "RemoveContainer" containerID="0b9253d95096019e99e4a9ccdb67c57f96ad94f13347d910d06f2d585d0c4e1d" Oct 31 04:11:26.262512 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.260278 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862a6eb4-x6zjv\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:26.262512 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.260406 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlhz\" (UniqueName: \"kubernetes.io/projected/ca8d3fac-50c1-4d10-9138-35446c73f10c-kube-api-access-zqlhz\") pod \"maas-api-revoke-sub-862a6eb4-x6zjv\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:26.262512 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.260443 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862a6eb4-x6zjv\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:26.262954 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.262925 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862a6eb4-x6zjv\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:26.264515 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.264492 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862a6eb4-x6zjv\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:26.281121 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.281096 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:26.319706 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.319681 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlhz\" (UniqueName: \"kubernetes.io/projected/ca8d3fac-50c1-4d10-9138-35446c73f10c-kube-api-access-zqlhz\") pod \"maas-api-revoke-sub-862a6eb4-x6zjv\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:26.386034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.386000 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:26.669386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.669353 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr"] Oct 31 04:11:26.673477 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:11:26.673443 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c85669_cd8d_4766_b519_e2294dc5fc20.slice/crio-12519759f2d0c2484df7e1b46bac3c34c269670419896e755ad7a8bc13d9546c WatchSource:0}: Error finding container 12519759f2d0c2484df7e1b46bac3c34c269670419896e755ad7a8bc13d9546c: Status 404 returned error can't find the container with id 12519759f2d0c2484df7e1b46bac3c34c269670419896e755ad7a8bc13d9546c Oct 31 04:11:26.744850 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.744815 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" event={"ID":"58c85669-cd8d-4766-b519-e2294dc5fc20","Type":"ContainerStarted","Data":"12519759f2d0c2484df7e1b46bac3c34c269670419896e755ad7a8bc13d9546c"} Oct 31 04:11:26.781410 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:26.781379 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv"] Oct 31 04:11:26.785733 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:11:26.785706 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8d3fac_50c1_4d10_9138_35446c73f10c.slice/crio-cd9be92ecc59b25b9e371aed576a01aa2ddcd7c19e7621662294679a16b366bf WatchSource:0}: Error finding container cd9be92ecc59b25b9e371aed576a01aa2ddcd7c19e7621662294679a16b366bf: Status 404 returned error can't find the container with id cd9be92ecc59b25b9e371aed576a01aa2ddcd7c19e7621662294679a16b366bf Oct 31 04:11:27.762075 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:27.762039 4426 generic.go:356] "Generic (PLEG): container finished" podID="58c85669-cd8d-4766-b519-e2294dc5fc20" containerID="54e269a0e2ec8cd41e7eddfb373b00d6ee5f4352237610913192298b9142831f" exitCode=0 Oct 31 04:11:27.762488 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:27.762161 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" event={"ID":"58c85669-cd8d-4766-b519-e2294dc5fc20","Type":"ContainerDied","Data":"54e269a0e2ec8cd41e7eddfb373b00d6ee5f4352237610913192298b9142831f"} Oct 31 04:11:27.763976 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:27.763954 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" event={"ID":"ca8d3fac-50c1-4d10-9138-35446c73f10c","Type":"ContainerStarted","Data":"d3fde62f8d5ea893a6e83511ef69be8387e5c99b7f3edbf746f0eb18cdf626ba"} Oct 31 04:11:27.764085 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:27.763998 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" event={"ID":"ca8d3fac-50c1-4d10-9138-35446c73f10c","Type":"ContainerStarted","Data":"cd9be92ecc59b25b9e371aed576a01aa2ddcd7c19e7621662294679a16b366bf"} Oct 31 04:11:27.985770 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:27.985699 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" podStartSLOduration=2.985678777 podStartE2EDuration="2.985678777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:27.981522365 +0000 UTC m=+2584.982147592" watchObservedRunningTime="2026-10-31 04:11:27.985678777 +0000 UTC m=+2584.986304006" Oct 31 04:11:28.397399 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:28.397344 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" podUID="07b122da-b857-45bb-979d-696f42767f4c" containerName="payload-pre-processing" containerID="cri-o://ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597" gracePeriod=25 Oct 31 04:11:28.773378 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:28.773286 4426 generic.go:356] "Generic (PLEG): container finished" podID="ca8d3fac-50c1-4d10-9138-35446c73f10c" containerID="d3fde62f8d5ea893a6e83511ef69be8387e5c99b7f3edbf746f0eb18cdf626ba" exitCode=0 Oct 31 04:11:28.773378 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:28.773341 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" event={"ID":"ca8d3fac-50c1-4d10-9138-35446c73f10c","Type":"ContainerDied","Data":"d3fde62f8d5ea893a6e83511ef69be8387e5c99b7f3edbf746f0eb18cdf626ba"} Oct 31 04:11:28.924817 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:28.924788 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:29.001709 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.001665 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-openshift-ca\") pod \"58c85669-cd8d-4766-b519-e2294dc5fc20\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " Oct 31 04:11:29.001894 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.001780 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c85669-cd8d-4766-b519-e2294dc5fc20-kube-api-access-vdv2p\" (UniqueName: \"kubernetes.io/projected/58c85669-cd8d-4766-b519-e2294dc5fc20-kube-api-access-vdv2p\") pod \"58c85669-cd8d-4766-b519-e2294dc5fc20\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " Oct 31 04:11:29.001894 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.001821 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-xks-ca\") pod \"58c85669-cd8d-4766-b519-e2294dc5fc20\" (UID: \"58c85669-cd8d-4766-b519-e2294dc5fc20\") " Oct 31 04:11:29.002109 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.002080 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-openshift-ca" pod "58c85669-cd8d-4766-b519-e2294dc5fc20" (UID: "58c85669-cd8d-4766-b519-e2294dc5fc20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:29.003896 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.003870 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-xks-ca" pod "58c85669-cd8d-4766-b519-e2294dc5fc20" (UID: "58c85669-cd8d-4766-b519-e2294dc5fc20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:29.004078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.004018 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c85669-cd8d-4766-b519-e2294dc5fc20-kube-api-access-vdv2p" pod "58c85669-cd8d-4766-b519-e2294dc5fc20" (UID: "58c85669-cd8d-4766-b519-e2294dc5fc20"). InnerVolumeSpecName "kube-api-access-vdv2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:29.103498 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.103412 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:29.103498 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.103446 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdv2p\" (UniqueName: \"kubernetes.io/projected/58c85669-cd8d-4766-b519-e2294dc5fc20-kube-api-access-vdv2p\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:29.103498 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.103457 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c85669-cd8d-4766-b519-e2294dc5fc20-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:29.782630 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.782599 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" Oct 31 04:11:29.783095 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.782632 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr" event={"ID":"58c85669-cd8d-4766-b519-e2294dc5fc20","Type":"ContainerDied","Data":"12519759f2d0c2484df7e1b46bac3c34c269670419896e755ad7a8bc13d9546c"} Oct 31 04:11:29.783095 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.782686 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12519759f2d0c2484df7e1b46bac3c34c269670419896e755ad7a8bc13d9546c" Oct 31 04:11:29.944299 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:29.944270 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:30.115338 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.115244 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-openshift-ca\") pod \"ca8d3fac-50c1-4d10-9138-35446c73f10c\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " Oct 31 04:11:30.115338 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.115310 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8d3fac-50c1-4d10-9138-35446c73f10c-kube-api-access-zqlhz\" (UniqueName: \"kubernetes.io/projected/ca8d3fac-50c1-4d10-9138-35446c73f10c-kube-api-access-zqlhz\") pod \"ca8d3fac-50c1-4d10-9138-35446c73f10c\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " Oct 31 04:11:30.115648 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.115432 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-xks-ca\") pod \"ca8d3fac-50c1-4d10-9138-35446c73f10c\" (UID: \"ca8d3fac-50c1-4d10-9138-35446c73f10c\") " Oct 31 04:11:30.115724 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.115673 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-openshift-ca" pod "ca8d3fac-50c1-4d10-9138-35446c73f10c" (UID: "ca8d3fac-50c1-4d10-9138-35446c73f10c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:30.115947 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.115923 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:30.117647 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.117613 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-xks-ca" pod "ca8d3fac-50c1-4d10-9138-35446c73f10c" (UID: "ca8d3fac-50c1-4d10-9138-35446c73f10c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:30.117647 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.117636 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8d3fac-50c1-4d10-9138-35446c73f10c-kube-api-access-zqlhz" pod "ca8d3fac-50c1-4d10-9138-35446c73f10c" (UID: "ca8d3fac-50c1-4d10-9138-35446c73f10c"). InnerVolumeSpecName "kube-api-access-zqlhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:30.217762 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.217716 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqlhz\" (UniqueName: \"kubernetes.io/projected/ca8d3fac-50c1-4d10-9138-35446c73f10c-kube-api-access-zqlhz\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:30.217762 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.217757 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8d3fac-50c1-4d10-9138-35446c73f10c-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:30.605877 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.605808 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" podUID="e061f010-dc83-4cb6-ae96-ce0a382a935d" containerName="payload-processing" containerID="cri-o://69a5545fcbb57850fd705706742f417253a85fe45c7de894b53c78b9d1dd2ded" gracePeriod=25 Oct 31 04:11:30.705229 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.705161 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" podUID="901174d2-5cca-4e9b-8d17-bfb128156b2c" containerName="payload-pre-processing" containerID="cri-o://be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0" gracePeriod=25 Oct 31 04:11:30.794447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.794416 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" Oct 31 04:11:30.794849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.794432 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv" event={"ID":"ca8d3fac-50c1-4d10-9138-35446c73f10c","Type":"ContainerDied","Data":"cd9be92ecc59b25b9e371aed576a01aa2ddcd7c19e7621662294679a16b366bf"} Oct 31 04:11:30.794849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.794526 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd9be92ecc59b25b9e371aed576a01aa2ddcd7c19e7621662294679a16b366bf" Oct 31 04:11:30.796776 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.796744 4426 generic.go:356] "Generic (PLEG): container finished" podID="e061f010-dc83-4cb6-ae96-ce0a382a935d" containerID="69a5545fcbb57850fd705706742f417253a85fe45c7de894b53c78b9d1dd2ded" exitCode=0 Oct 31 04:11:30.796908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.796814 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" event={"ID":"e061f010-dc83-4cb6-ae96-ce0a382a935d","Type":"ContainerDied","Data":"69a5545fcbb57850fd705706742f417253a85fe45c7de894b53c78b9d1dd2ded"} Oct 31 04:11:30.852985 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.852908 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:11:30.907071 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.907039 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd"] Oct 31 04:11:30.907295 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:30.907269 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" podUID="f825f3ae-e9e6-4358-aca1-9e575b0d6d91" containerName="discovery" containerID="cri-o://c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5" gracePeriod=30 Oct 31 04:11:31.039431 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.039398 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:11:31.049570 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.049534 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e061f010-dc83-4cb6-ae96-ce0a382a935d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e061f010-dc83-4cb6-ae96-ce0a382a935d-plugins-config-volume\") pod \"e061f010-dc83-4cb6-ae96-ce0a382a935d\" (UID: \"e061f010-dc83-4cb6-ae96-ce0a382a935d\") " Oct 31 04:11:31.049762 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.049597 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e061f010-dc83-4cb6-ae96-ce0a382a935d-kube-api-access-7gpfk\" (UniqueName: \"kubernetes.io/projected/e061f010-dc83-4cb6-ae96-ce0a382a935d-kube-api-access-7gpfk\") pod \"e061f010-dc83-4cb6-ae96-ce0a382a935d\" (UID: \"e061f010-dc83-4cb6-ae96-ce0a382a935d\") " Oct 31 04:11:31.050363 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.050331 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e061f010-dc83-4cb6-ae96-ce0a382a935d-plugins-config-volume" pod "e061f010-dc83-4cb6-ae96-ce0a382a935d" (UID: "e061f010-dc83-4cb6-ae96-ce0a382a935d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:31.052045 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.052015 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e061f010-dc83-4cb6-ae96-ce0a382a935d-kube-api-access-7gpfk" pod "e061f010-dc83-4cb6-ae96-ce0a382a935d" (UID: "e061f010-dc83-4cb6-ae96-ce0a382a935d"). InnerVolumeSpecName "kube-api-access-7gpfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:31.150439 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.150403 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901174d2-5cca-4e9b-8d17-bfb128156b2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901174d2-5cca-4e9b-8d17-bfb128156b2c-plugins-config-volume\") pod \"901174d2-5cca-4e9b-8d17-bfb128156b2c\" (UID: \"901174d2-5cca-4e9b-8d17-bfb128156b2c\") " Oct 31 04:11:31.150808 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.150782 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901174d2-5cca-4e9b-8d17-bfb128156b2c-kube-api-access-b5kpk\" (UniqueName: \"kubernetes.io/projected/901174d2-5cca-4e9b-8d17-bfb128156b2c-kube-api-access-b5kpk\") pod \"901174d2-5cca-4e9b-8d17-bfb128156b2c\" (UID: \"901174d2-5cca-4e9b-8d17-bfb128156b2c\") " Oct 31 04:11:31.150913 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.150891 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901174d2-5cca-4e9b-8d17-bfb128156b2c-plugins-config-volume" pod "901174d2-5cca-4e9b-8d17-bfb128156b2c" (UID: "901174d2-5cca-4e9b-8d17-bfb128156b2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:31.151204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.151184 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e061f010-dc83-4cb6-ae96-ce0a382a935d-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.151282 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.151210 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gpfk\" (UniqueName: \"kubernetes.io/projected/e061f010-dc83-4cb6-ae96-ce0a382a935d-kube-api-access-7gpfk\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.151282 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.151226 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901174d2-5cca-4e9b-8d17-bfb128156b2c-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.153434 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.153411 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901174d2-5cca-4e9b-8d17-bfb128156b2c-kube-api-access-b5kpk" pod "901174d2-5cca-4e9b-8d17-bfb128156b2c" (UID: "901174d2-5cca-4e9b-8d17-bfb128156b2c"). InnerVolumeSpecName "kube-api-access-b5kpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:31.178774 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.178747 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 04:11:31.251642 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.251606 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-cacerts\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-cacerts\") pod \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " Oct 31 04:11:31.251866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.251653 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-local-certs\") pod \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " Oct 31 04:11:31.251866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.251690 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-kube-api-access-z9bsd\" (UniqueName: \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-kube-api-access-z9bsd\") pod \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " Oct 31 04:11:31.251866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.251732 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-token\" (UniqueName: \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-token\") pod \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " Oct 31 04:11:31.251866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.251762 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-ca-configmap\") pod \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " Oct 31 04:11:31.251866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.251806 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-dns-cert\") pod \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " Oct 31 04:11:31.252401 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.252373 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-kubeconfig\") pod \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\" (UID: \"f825f3ae-e9e6-4358-aca1-9e575b0d6d91\") " Oct 31 04:11:31.252593 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.252372 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-ca-configmap" pod "f825f3ae-e9e6-4358-aca1-9e575b0d6d91" (UID: "f825f3ae-e9e6-4358-aca1-9e575b0d6d91"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:31.252980 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.252956 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5kpk\" (UniqueName: \"kubernetes.io/projected/901174d2-5cca-4e9b-8d17-bfb128156b2c-kube-api-access-b5kpk\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.253079 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.252987 4426 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-ca-configmap\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.254077 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.254050 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-cacerts" pod "f825f3ae-e9e6-4358-aca1-9e575b0d6d91" (UID: "f825f3ae-e9e6-4358-aca1-9e575b0d6d91"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:31.254357 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.254336 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-local-certs" pod "f825f3ae-e9e6-4358-aca1-9e575b0d6d91" (UID: "f825f3ae-e9e6-4358-aca1-9e575b0d6d91"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:31.254421 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.254390 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-kubeconfig" pod "f825f3ae-e9e6-4358-aca1-9e575b0d6d91" (UID: "f825f3ae-e9e6-4358-aca1-9e575b0d6d91"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:31.254618 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.254599 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-token" pod "f825f3ae-e9e6-4358-aca1-9e575b0d6d91" (UID: "f825f3ae-e9e6-4358-aca1-9e575b0d6d91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:31.254679 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.254629 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-kube-api-access-z9bsd" pod "f825f3ae-e9e6-4358-aca1-9e575b0d6d91" (UID: "f825f3ae-e9e6-4358-aca1-9e575b0d6d91"). InnerVolumeSpecName "kube-api-access-z9bsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:31.254736 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.254720 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-dns-cert" pod "f825f3ae-e9e6-4358-aca1-9e575b0d6d91" (UID: "f825f3ae-e9e6-4358-aca1-9e575b0d6d91"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:31.354578 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.354542 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.354578 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.354578 4426 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-csr-dns-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.354767 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.354592 4426 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-istio-kubeconfig\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.354767 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.354605 4426 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-cacerts\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.354767 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.354618 4426 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-local-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.354767 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.354630 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9bsd\" (UniqueName: \"kubernetes.io/projected/f825f3ae-e9e6-4358-aca1-9e575b0d6d91-kube-api-access-z9bsd\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.389036 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.388995 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" containerID="cri-o://b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8" gracePeriod=45 Oct 31 04:11:31.413825 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.413792 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" podUID="07b122da-b857-45bb-979d-696f42767f4c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:11:31.627649 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.627621 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 04:11:31.657930 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.657840 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-model-cache\") pod \"42350f0d-272a-48cf-bfbf-9c60ffa55221\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " Oct 31 04:11:31.657930 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.657902 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42350f0d-272a-48cf-bfbf-9c60ffa55221-tls-certs\" (UniqueName: \"kubernetes.io/secret/42350f0d-272a-48cf-bfbf-9c60ffa55221-tls-certs\") pod \"42350f0d-272a-48cf-bfbf-9c60ffa55221\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " Oct 31 04:11:31.658206 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.657948 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-tmp-dir\") pod \"42350f0d-272a-48cf-bfbf-9c60ffa55221\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " Oct 31 04:11:31.658206 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.658037 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-model-cache" pod "42350f0d-272a-48cf-bfbf-9c60ffa55221" (UID: "42350f0d-272a-48cf-bfbf-9c60ffa55221"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:31.658206 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.658090 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-home\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-home\") pod \"42350f0d-272a-48cf-bfbf-9c60ffa55221\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " Oct 31 04:11:31.658206 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.658172 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42350f0d-272a-48cf-bfbf-9c60ffa55221-kube-api-access-wn225\" (UniqueName: \"kubernetes.io/projected/42350f0d-272a-48cf-bfbf-9c60ffa55221-kube-api-access-wn225\") pod \"42350f0d-272a-48cf-bfbf-9c60ffa55221\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " Oct 31 04:11:31.658206 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.658203 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-dshm\") pod \"42350f0d-272a-48cf-bfbf-9c60ffa55221\" (UID: \"42350f0d-272a-48cf-bfbf-9c60ffa55221\") " Oct 31 04:11:31.658487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.658245 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-tmp-dir" pod "42350f0d-272a-48cf-bfbf-9c60ffa55221" (UID: "42350f0d-272a-48cf-bfbf-9c60ffa55221"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:31.658487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.658305 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-home" pod "42350f0d-272a-48cf-bfbf-9c60ffa55221" (UID: "42350f0d-272a-48cf-bfbf-9c60ffa55221"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:31.658487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.658479 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.658640 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.658509 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.658640 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.658529 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.660422 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.660395 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-dshm" pod "42350f0d-272a-48cf-bfbf-9c60ffa55221" (UID: "42350f0d-272a-48cf-bfbf-9c60ffa55221"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:31.660534 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.660472 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42350f0d-272a-48cf-bfbf-9c60ffa55221-kube-api-access-wn225" pod "42350f0d-272a-48cf-bfbf-9c60ffa55221" (UID: "42350f0d-272a-48cf-bfbf-9c60ffa55221"). InnerVolumeSpecName "kube-api-access-wn225". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:31.660534 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.660478 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42350f0d-272a-48cf-bfbf-9c60ffa55221-tls-certs" pod "42350f0d-272a-48cf-bfbf-9c60ffa55221" (UID: "42350f0d-272a-48cf-bfbf-9c60ffa55221"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:31.759474 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.759434 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn225\" (UniqueName: \"kubernetes.io/projected/42350f0d-272a-48cf-bfbf-9c60ffa55221-kube-api-access-wn225\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.759474 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.759477 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42350f0d-272a-48cf-bfbf-9c60ffa55221-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.759677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.759493 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42350f0d-272a-48cf-bfbf-9c60ffa55221-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:31.806115 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.806074 4426 generic.go:356] "Generic (PLEG): container finished" podID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerID="b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8" exitCode=0 Oct 31 04:11:31.806630 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.806117 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" Oct 31 04:11:31.806630 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.806161 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" event={"ID":"42350f0d-272a-48cf-bfbf-9c60ffa55221","Type":"ContainerDied","Data":"b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8"} Oct 31 04:11:31.806630 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.806219 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d" event={"ID":"42350f0d-272a-48cf-bfbf-9c60ffa55221","Type":"ContainerDied","Data":"915f81d00de30edaef70330efa8b6aa274839c33ab238bb6168bbef30be47a9b"} Oct 31 04:11:31.806630 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.806246 4426 scope.go:122] "RemoveContainer" containerID="b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8" Oct 31 04:11:31.808097 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.808077 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" Oct 31 04:11:31.808311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.808086 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf" event={"ID":"e061f010-dc83-4cb6-ae96-ce0a382a935d","Type":"ContainerDied","Data":"f09df8741e926e5a0e44a73c4210112ef4abf85bfeb4c14f6e4ac6bd5aa9c3ac"} Oct 31 04:11:31.809904 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.809878 4426 generic.go:356] "Generic (PLEG): container finished" podID="901174d2-5cca-4e9b-8d17-bfb128156b2c" containerID="be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0" exitCode=0 Oct 31 04:11:31.811291 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.809936 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" Oct 31 04:11:31.811291 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.809959 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" event={"ID":"901174d2-5cca-4e9b-8d17-bfb128156b2c","Type":"ContainerDied","Data":"be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0"} Oct 31 04:11:31.811291 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.810021 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw" event={"ID":"901174d2-5cca-4e9b-8d17-bfb128156b2c","Type":"ContainerDied","Data":"947d8ede5bd92e87d28f741898fd6eab5a6baa9f060a58df36423caa3ee6c67b"} Oct 31 04:11:31.812453 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.812420 4426 generic.go:356] "Generic (PLEG): container finished" podID="f825f3ae-e9e6-4358-aca1-9e575b0d6d91" containerID="c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5" exitCode=0 Oct 31 04:11:31.812567 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.812519 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" Oct 31 04:11:31.812567 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.812540 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" event={"ID":"f825f3ae-e9e6-4358-aca1-9e575b0d6d91","Type":"ContainerDied","Data":"c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5"} Oct 31 04:11:31.812685 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.812573 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd" event={"ID":"f825f3ae-e9e6-4358-aca1-9e575b0d6d91","Type":"ContainerDied","Data":"ab913b04786952095b3bc7f4f92622f3c8387be0bfc664dc7bcada47265e274c"} Oct 31 04:11:31.820950 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.820928 4426 scope.go:122] "RemoveContainer" containerID="b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8" Oct 31 04:11:31.821250 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:11:31.821229 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8\": container with ID starting with b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8 not found: ID does not exist" containerID="b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8" Oct 31 04:11:31.821353 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.821257 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8"} err="failed to get container status \"b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8\": rpc error: code = NotFound desc = could not find container \"b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8\": container with ID starting with b83f13c7573454ce175d0755194edfba2d94244479344336d00a90c0d62245f8 not found: ID does not exist" Oct 31 04:11:31.821353 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.821275 4426 scope.go:122] "RemoveContainer" containerID="69a5545fcbb57850fd705706742f417253a85fe45c7de894b53c78b9d1dd2ded" Oct 31 04:11:31.831970 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.831816 4426 scope.go:122] "RemoveContainer" containerID="be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0" Oct 31 04:11:31.841025 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.841006 4426 scope.go:122] "RemoveContainer" containerID="be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0" Oct 31 04:11:31.841447 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:11:31.841421 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0\": container with ID starting with be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0 not found: ID does not exist" containerID="be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0" Oct 31 04:11:31.841510 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.841449 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0"} err="failed to get container status \"be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0\": rpc error: code = NotFound desc = could not find container \"be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0\": container with ID starting with be31d16420f585100732861504f8d7eb4e65b7954b6ffcd43eea1a82f3a082b0 not found: ID does not exist" Oct 31 04:11:31.841510 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.841468 4426 scope.go:122] "RemoveContainer" containerID="c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5" Oct 31 04:11:31.852656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.852562 4426 scope.go:122] "RemoveContainer" containerID="c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5" Oct 31 04:11:31.853050 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:11:31.853015 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5\": container with ID starting with c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5 not found: ID does not exist" containerID="c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5" Oct 31 04:11:31.853237 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.853061 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5"} err="failed to get container status \"c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5\": rpc error: code = NotFound desc = could not find container \"c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5\": container with ID starting with c06a1e4acbfb68b3ab2c4713ed26bc79a4da0fded0c2f608bfecf9a65f6234a5 not found: ID does not exist" Oct 31 04:11:31.967388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.967349 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw"] Oct 31 04:11:31.974677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:31.974638 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e81aea5-6cff4b79dc-779qw"] Oct 31 04:11:32.085480 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:32.085447 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d"] Oct 31 04:11:32.090577 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:32.090547 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e5b8f9/simulator-w5-e5b8f9-kserve-7dd76647c5-qxd5d"] Oct 31 04:11:32.231043 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:32.230957 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd"] Oct 31 04:11:32.262313 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:32.262270 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kwcsd"] Oct 31 04:11:32.363736 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:32.363693 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf"] Oct 31 04:11:32.367644 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:32.367614 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e81aea5-db74db9c5-9r7gf"] Oct 31 04:11:32.853660 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:11:32.853620 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:11:32.958611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:32.958578 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm"] Oct 31 04:11:32.961935 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:32.961907 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e81aea5-25kvm"] Oct 31 04:11:33.130635 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:33.130538 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" path="/var/lib/kubelet/pods/42350f0d-272a-48cf-bfbf-9c60ffa55221/volumes" Oct 31 04:11:33.130934 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:33.130918 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901174d2-5cca-4e9b-8d17-bfb128156b2c" path="/var/lib/kubelet/pods/901174d2-5cca-4e9b-8d17-bfb128156b2c/volumes" Oct 31 04:11:33.131325 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:33.131309 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e63911-ac83-45dc-bb91-64de3d67d4a5" path="/var/lib/kubelet/pods/d0e63911-ac83-45dc-bb91-64de3d67d4a5/volumes" Oct 31 04:11:33.131936 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:33.131920 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e061f010-dc83-4cb6-ae96-ce0a382a935d" path="/var/lib/kubelet/pods/e061f010-dc83-4cb6-ae96-ce0a382a935d/volumes" Oct 31 04:11:33.132357 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:33.132342 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f825f3ae-e9e6-4358-aca1-9e575b0d6d91" path="/var/lib/kubelet/pods/f825f3ae-e9e6-4358-aca1-9e575b0d6d91/volumes" Oct 31 04:11:38.966198 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:38.966114 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277"] Oct 31 04:11:39.013281 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:39.013247 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e5b8f9-9b277"] Oct 31 04:11:39.130523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:39.130478 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497e7052-450d-477c-9424-205a73356be7" path="/var/lib/kubelet/pods/497e7052-450d-477c-9424-205a73356be7/volumes" Oct 31 04:11:41.413773 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:41.413734 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" podUID="07b122da-b857-45bb-979d-696f42767f4c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:11:45.324266 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:45.324225 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:11:45.892883 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:45.892836 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm"] Oct 31 04:11:45.893343 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:45.893293 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" containerID="cri-o://3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3" gracePeriod=30 Oct 31 04:11:48.668234 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.668161 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 04:11:48.741447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.741407 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b122da-b857-45bb-979d-696f42767f4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b122da-b857-45bb-979d-696f42767f4c-plugins-config-volume\") pod \"07b122da-b857-45bb-979d-696f42767f4c\" (UID: \"07b122da-b857-45bb-979d-696f42767f4c\") " Oct 31 04:11:48.741628 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.741510 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b122da-b857-45bb-979d-696f42767f4c-kube-api-access-64gfv\" (UniqueName: \"kubernetes.io/projected/07b122da-b857-45bb-979d-696f42767f4c-kube-api-access-64gfv\") pod \"07b122da-b857-45bb-979d-696f42767f4c\" (UID: \"07b122da-b857-45bb-979d-696f42767f4c\") " Oct 31 04:11:48.741916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.741884 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b122da-b857-45bb-979d-696f42767f4c-plugins-config-volume" pod "07b122da-b857-45bb-979d-696f42767f4c" (UID: "07b122da-b857-45bb-979d-696f42767f4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:48.743660 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.743635 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b122da-b857-45bb-979d-696f42767f4c-kube-api-access-64gfv" pod "07b122da-b857-45bb-979d-696f42767f4c" (UID: "07b122da-b857-45bb-979d-696f42767f4c"). InnerVolumeSpecName "kube-api-access-64gfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:48.843372 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.843326 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64gfv\" (UniqueName: \"kubernetes.io/projected/07b122da-b857-45bb-979d-696f42767f4c-kube-api-access-64gfv\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:48.843372 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.843363 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b122da-b857-45bb-979d-696f42767f4c-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:48.950158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.950048 4426 generic.go:356] "Generic (PLEG): container finished" podID="07b122da-b857-45bb-979d-696f42767f4c" containerID="ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597" exitCode=0 Oct 31 04:11:48.950158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.950106 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" Oct 31 04:11:48.950360 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.950159 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" event={"ID":"07b122da-b857-45bb-979d-696f42767f4c","Type":"ContainerDied","Data":"ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597"} Oct 31 04:11:48.950360 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.950210 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8" event={"ID":"07b122da-b857-45bb-979d-696f42767f4c","Type":"ContainerDied","Data":"10d8fdf0ba5585729c30ae359071c1d3c4ea44583d8c963de3daadb65eba8ca8"} Oct 31 04:11:48.950360 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.950236 4426 scope.go:122] "RemoveContainer" containerID="ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597" Oct 31 04:11:48.960759 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.960736 4426 scope.go:122] "RemoveContainer" containerID="ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597" Oct 31 04:11:48.961071 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:11:48.961051 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597\": container with ID starting with ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597 not found: ID does not exist" containerID="ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597" Oct 31 04:11:48.961158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:48.961080 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597"} err="failed to get container status \"ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597\": rpc error: code = NotFound desc = could not find container \"ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597\": container with ID starting with ae2c8680a3053804bc35daa2230c385f15b9e3ca46e6343263e81d5cbbc51597 not found: ID does not exist" Oct 31 04:11:49.106172 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:49.106137 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8"] Oct 31 04:11:49.109250 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:49.109226 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e5b8f9-6c98867d4c-grml8"] Oct 31 04:11:49.128316 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:49.128292 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b122da-b857-45bb-979d-696f42767f4c" path="/var/lib/kubelet/pods/07b122da-b857-45bb-979d-696f42767f4c/volumes" Oct 31 04:11:51.164545 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.164481 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 04:11:51.270248 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.270213 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-kube-api-access-k4wpf\" (UniqueName: \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-kube-api-access-k4wpf\") pod \"85150ab8-f8d1-40d2-b540-0f65db33199b\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " Oct 31 04:11:51.270248 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.270249 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85150ab8-f8d1-40d2-b540-0f65db33199b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85150ab8-f8d1-40d2-b540-0f65db33199b-istiod-ca-cert\") pod \"85150ab8-f8d1-40d2-b540-0f65db33199b\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " Oct 31 04:11:51.270518 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.270285 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-envoy\") pod \"85150ab8-f8d1-40d2-b540-0f65db33199b\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " Oct 31 04:11:51.270518 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.270330 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-credential-socket\") pod \"85150ab8-f8d1-40d2-b540-0f65db33199b\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " Oct 31 04:11:51.270518 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.270424 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-socket\") pod \"85150ab8-f8d1-40d2-b540-0f65db33199b\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " Oct 31 04:11:51.270518 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.270462 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-podinfo\") pod \"85150ab8-f8d1-40d2-b540-0f65db33199b\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " Oct 31 04:11:51.270518 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.270512 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-token\" (UniqueName: \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-token\") pod \"85150ab8-f8d1-40d2-b540-0f65db33199b\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " Oct 31 04:11:51.270781 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.270543 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-certs\") pod \"85150ab8-f8d1-40d2-b540-0f65db33199b\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " Oct 31 04:11:51.270781 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.270572 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-data\") pod \"85150ab8-f8d1-40d2-b540-0f65db33199b\" (UID: \"85150ab8-f8d1-40d2-b540-0f65db33199b\") " Oct 31 04:11:51.270781 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.270716 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-credential-socket" pod "85150ab8-f8d1-40d2-b540-0f65db33199b" (UID: "85150ab8-f8d1-40d2-b540-0f65db33199b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.270781 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.270739 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85150ab8-f8d1-40d2-b540-0f65db33199b-istiod-ca-cert" pod "85150ab8-f8d1-40d2-b540-0f65db33199b" (UID: "85150ab8-f8d1-40d2-b540-0f65db33199b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:51.271067 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.271039 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-socket" pod "85150ab8-f8d1-40d2-b540-0f65db33199b" (UID: "85150ab8-f8d1-40d2-b540-0f65db33199b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.271216 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.271194 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.271292 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.271222 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.271292 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.271240 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85150ab8-f8d1-40d2-b540-0f65db33199b-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.271292 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.271197 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-certs" pod "85150ab8-f8d1-40d2-b540-0f65db33199b" (UID: "85150ab8-f8d1-40d2-b540-0f65db33199b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.271450 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.271371 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-data" pod "85150ab8-f8d1-40d2-b540-0f65db33199b" (UID: "85150ab8-f8d1-40d2-b540-0f65db33199b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.272871 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.272837 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-kube-api-access-k4wpf" pod "85150ab8-f8d1-40d2-b540-0f65db33199b" (UID: "85150ab8-f8d1-40d2-b540-0f65db33199b"). InnerVolumeSpecName "kube-api-access-k4wpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:51.273074 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.273046 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-envoy" pod "85150ab8-f8d1-40d2-b540-0f65db33199b" (UID: "85150ab8-f8d1-40d2-b540-0f65db33199b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.273074 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.273059 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-podinfo" pod "85150ab8-f8d1-40d2-b540-0f65db33199b" (UID: "85150ab8-f8d1-40d2-b540-0f65db33199b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:11:51.273462 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.273439 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-token" pod "85150ab8-f8d1-40d2-b540-0f65db33199b" (UID: "85150ab8-f8d1-40d2-b540-0f65db33199b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:51.371978 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.371940 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.371978 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.371972 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.371978 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.371986 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.372354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.372001 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4wpf\" (UniqueName: \"kubernetes.io/projected/85150ab8-f8d1-40d2-b540-0f65db33199b-kube-api-access-k4wpf\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.372354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.372013 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.372354 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.372021 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85150ab8-f8d1-40d2-b540-0f65db33199b-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.980266 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.980229 4426 generic.go:356] "Generic (PLEG): container finished" podID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerID="3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3" exitCode=0 Oct 31 04:11:51.980447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.980294 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" Oct 31 04:11:51.980447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.980316 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" event={"ID":"85150ab8-f8d1-40d2-b540-0f65db33199b","Type":"ContainerDied","Data":"3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3"} Oct 31 04:11:51.980447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.980407 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm" event={"ID":"85150ab8-f8d1-40d2-b540-0f65db33199b","Type":"ContainerDied","Data":"34e05b09397dcb03b1e9863c7f9cd426b230c3cd915021c081f15f27f9c6b3d0"} Oct 31 04:11:51.980447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.980437 4426 scope.go:122] "RemoveContainer" containerID="3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3" Oct 31 04:11:51.992406 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:51.992386 4426 scope.go:122] "RemoveContainer" containerID="ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675" Oct 31 04:11:52.002814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:52.002796 4426 scope.go:122] "RemoveContainer" containerID="3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3" Oct 31 04:11:52.003108 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:11:52.003085 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3\": container with ID starting with 3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3 not found: ID does not exist" containerID="3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3" Oct 31 04:11:52.003281 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:52.003119 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3"} err="failed to get container status \"3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3\": rpc error: code = NotFound desc = could not find container \"3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3\": container with ID starting with 3f3a0552d67086512d217f9fe1b22a46fe588b62cf84d6173c8f2ae534afead3 not found: ID does not exist" Oct 31 04:11:52.003281 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:52.003172 4426 scope.go:122] "RemoveContainer" containerID="ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675" Oct 31 04:11:52.003468 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:11:52.003450 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675\": container with ID starting with ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675 not found: ID does not exist" containerID="ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675" Oct 31 04:11:52.003522 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:52.003478 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675"} err="failed to get container status \"ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675\": rpc error: code = NotFound desc = could not find container \"ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675\": container with ID starting with ddb0957feac1ce94a72102e60d23cb8e0ab3342d52abba09b70e9b07d6511675 not found: ID does not exist" Oct 31 04:11:52.183463 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:52.183430 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm"] Oct 31 04:11:52.229941 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:52.229907 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-412ba5-openshift-default-554dbd8599-qxlvm"] Oct 31 04:11:53.128566 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:11:53.128533 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" path="/var/lib/kubelet/pods/85150ab8-f8d1-40d2-b540-0f65db33199b/volumes" Oct 31 04:12:02.845336 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:12:02.845297 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:12:04.048040 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:04.048001 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz"] Oct 31 04:12:04.094452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:04.094409 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs"] Oct 31 04:12:04.097569 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:04.097536 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32bb28c5-6t2lz"] Oct 31 04:12:04.100348 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:04.100319 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b41da457-nj2gs"] Oct 31 04:12:05.128709 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:05.128669 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f04685-9b38-494b-a38d-560e2411be34" path="/var/lib/kubelet/pods/55f04685-9b38-494b-a38d-560e2411be34/volumes" Oct 31 04:12:05.129117 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:05.129108 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73bb3d25-ee33-4991-88ec-9f84ff8de89b" path="/var/lib/kubelet/pods/73bb3d25-ee33-4991-88ec-9f84ff8de89b/volumes" Oct 31 04:12:26.324970 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:26.324934 4426 scope.go:122] "RemoveContainer" containerID="d787a9f62171f777c6cd9e41df538aebbb0813c2f6f04c5882eb27a6fc6ff375" Oct 31 04:12:27.944239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:27.944196 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt"] Oct 31 04:12:32.846417 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:12:32.846375 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:12:34.957200 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:34.957159 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl"] Oct 31 04:12:37.045681 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.045596 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl"] Oct 31 04:12:37.092158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.092110 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849dc88c-xbxsl"] Oct 31 04:12:37.128710 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.128677 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba6ac38-84db-419c-899f-af8ae6904dc1" path="/var/lib/kubelet/pods/aba6ac38-84db-419c-899f-af8ae6904dc1/volumes" Oct 31 04:12:37.993779 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.993745 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g"] Oct 31 04:12:37.994365 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994337 4426 cpu_manager.go:414] "containerMap: removing container" podUID="58c85669-cd8d-4766-b519-e2294dc5fc20" containerName="revoke-keys" Oct 31 04:12:37.994521 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994424 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c85669-cd8d-4766-b519-e2294dc5fc20" containerName="revoke-keys" Oct 31 04:12:37.994521 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994451 4426 cpu_manager.go:414] "containerMap: removing container" podUID="901174d2-5cca-4e9b-8d17-bfb128156b2c" containerName="payload-pre-processing" Oct 31 04:12:37.994521 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994471 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901174d2-5cca-4e9b-8d17-bfb128156b2c" containerName="payload-pre-processing" Oct 31 04:12:37.994521 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994497 4426 cpu_manager.go:414] "containerMap: removing container" podUID="f825f3ae-e9e6-4358-aca1-9e575b0d6d91" containerName="discovery" Oct 31 04:12:37.994521 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994506 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f825f3ae-e9e6-4358-aca1-9e575b0d6d91" containerName="discovery" Oct 31 04:12:37.994521 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994519 4426 cpu_manager.go:414] "containerMap: removing container" podUID="ca8d3fac-50c1-4d10-9138-35446c73f10c" containerName="revoke-keys" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994526 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8d3fac-50c1-4d10-9138-35446c73f10c" containerName="revoke-keys" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994541 4426 cpu_manager.go:414] "containerMap: removing container" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994548 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994569 4426 cpu_manager.go:414] "containerMap: removing container" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994577 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994598 4426 cpu_manager.go:414] "containerMap: removing container" podUID="07b122da-b857-45bb-979d-696f42767f4c" containerName="payload-pre-processing" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994606 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b122da-b857-45bb-979d-696f42767f4c" containerName="payload-pre-processing" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994627 4426 cpu_manager.go:414] "containerMap: removing container" podUID="e061f010-dc83-4cb6-ae96-ce0a382a935d" containerName="payload-processing" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994635 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e061f010-dc83-4cb6-ae96-ce0a382a935d" containerName="payload-processing" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994766 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b122da-b857-45bb-979d-696f42767f4c" containerName="payload-pre-processing" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994779 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="e061f010-dc83-4cb6-ae96-ce0a382a935d" containerName="payload-processing" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994787 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994800 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c85669-cd8d-4766-b519-e2294dc5fc20" containerName="revoke-keys" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994808 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="901174d2-5cca-4e9b-8d17-bfb128156b2c" containerName="payload-pre-processing" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994819 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="f825f3ae-e9e6-4358-aca1-9e575b0d6d91" containerName="discovery" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994827 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8d3fac-50c1-4d10-9138-35446c73f10c" containerName="revoke-keys" Oct 31 04:12:37.994851 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:37.994837 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="42350f0d-272a-48cf-bfbf-9c60ffa55221" containerName="main" Oct 31 04:12:38.000320 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.000294 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:38.047241 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.047203 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq"] Oct 31 04:12:38.091895 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.091856 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:12:38.091895 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.091870 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:12:38.094509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.094486 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:12:38.095499 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.095473 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d3b696c-cxsvq"] Oct 31 04:12:38.100393 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.100361 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g"] Oct 31 04:12:38.132902 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.132862 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:12:38.164823 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.164784 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7a6243-4lm9g\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:38.165040 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.164851 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svgh8\" (UniqueName: \"kubernetes.io/projected/4d3609f5-449f-4506-ae90-2fceb42b9af5-kube-api-access-svgh8\") pod \"maas-api-revoke-sub-df7a6243-4lm9g\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:38.165040 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.164965 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7a6243-4lm9g\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:38.266203 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.266076 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svgh8\" (UniqueName: \"kubernetes.io/projected/4d3609f5-449f-4506-ae90-2fceb42b9af5-kube-api-access-svgh8\") pod \"maas-api-revoke-sub-df7a6243-4lm9g\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:38.266370 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.266206 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7a6243-4lm9g\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:38.266370 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.266358 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7a6243-4lm9g\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:38.267083 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.267059 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7a6243-4lm9g\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:38.268663 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.268638 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7a6243-4lm9g\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:38.324026 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.323989 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svgh8\" (UniqueName: \"kubernetes.io/projected/4d3609f5-449f-4506-ae90-2fceb42b9af5-kube-api-access-svgh8\") pod \"maas-api-revoke-sub-df7a6243-4lm9g\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:38.614781 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.614689 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:38.993236 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:38.993195 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g"] Oct 31 04:12:38.997064 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:12:38.997025 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3609f5_449f_4506_ae90_2fceb42b9af5.slice/crio-9228aadfbb7e1badeb80b06106cac16c336a387512fcfa75fe02ea63c3ec2c4e WatchSource:0}: Error finding container 9228aadfbb7e1badeb80b06106cac16c336a387512fcfa75fe02ea63c3ec2c4e: Status 404 returned error can't find the container with id 9228aadfbb7e1badeb80b06106cac16c336a387512fcfa75fe02ea63c3ec2c4e Oct 31 04:12:39.129723 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:39.129684 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c46a697e-7668-4ea6-9e57-a726b4b8edd6" path="/var/lib/kubelet/pods/c46a697e-7668-4ea6-9e57-a726b4b8edd6/volumes" Oct 31 04:12:39.314945 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:39.314856 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" event={"ID":"4d3609f5-449f-4506-ae90-2fceb42b9af5","Type":"ContainerStarted","Data":"9228aadfbb7e1badeb80b06106cac16c336a387512fcfa75fe02ea63c3ec2c4e"} Oct 31 04:12:39.958361 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:39.958319 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" podUID="259d0ab8-20b4-4c18-9876-962926e794b9" containerName="payload-pre-processing" containerID="cri-o://1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8" gracePeriod=25 Oct 31 04:12:40.323931 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:40.323896 4426 generic.go:356] "Generic (PLEG): container finished" podID="4d3609f5-449f-4506-ae90-2fceb42b9af5" containerID="c18c09864b4becb88a5c273da7f8c3fc8d30631daf3df06ff56b9156c3493ec9" exitCode=0 Oct 31 04:12:40.324380 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:40.323986 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" event={"ID":"4d3609f5-449f-4506-ae90-2fceb42b9af5","Type":"ContainerDied","Data":"c18c09864b4becb88a5c273da7f8c3fc8d30631daf3df06ff56b9156c3493ec9"} Oct 31 04:12:41.479772 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:41.479744 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:41.599793 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:41.599710 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-xks-ca\") pod \"4d3609f5-449f-4506-ae90-2fceb42b9af5\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " Oct 31 04:12:41.599970 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:41.599803 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3609f5-449f-4506-ae90-2fceb42b9af5-kube-api-access-svgh8\" (UniqueName: \"kubernetes.io/projected/4d3609f5-449f-4506-ae90-2fceb42b9af5-kube-api-access-svgh8\") pod \"4d3609f5-449f-4506-ae90-2fceb42b9af5\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " Oct 31 04:12:41.599970 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:41.599869 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-openshift-ca\") pod \"4d3609f5-449f-4506-ae90-2fceb42b9af5\" (UID: \"4d3609f5-449f-4506-ae90-2fceb42b9af5\") " Oct 31 04:12:41.600358 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:41.600327 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-openshift-ca" pod "4d3609f5-449f-4506-ae90-2fceb42b9af5" (UID: "4d3609f5-449f-4506-ae90-2fceb42b9af5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:12:41.601962 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:41.601933 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-xks-ca" pod "4d3609f5-449f-4506-ae90-2fceb42b9af5" (UID: "4d3609f5-449f-4506-ae90-2fceb42b9af5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:12:41.602055 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:41.602026 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3609f5-449f-4506-ae90-2fceb42b9af5-kube-api-access-svgh8" pod "4d3609f5-449f-4506-ae90-2fceb42b9af5" (UID: "4d3609f5-449f-4506-ae90-2fceb42b9af5"). InnerVolumeSpecName "kube-api-access-svgh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:12:41.701172 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:41.701116 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:41.701172 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:41.701169 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3609f5-449f-4506-ae90-2fceb42b9af5-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:41.701392 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:41.701190 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svgh8\" (UniqueName: \"kubernetes.io/projected/4d3609f5-449f-4506-ae90-2fceb42b9af5-kube-api-access-svgh8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:42.340622 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:42.340591 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" Oct 31 04:12:42.340828 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:42.340583 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g" event={"ID":"4d3609f5-449f-4506-ae90-2fceb42b9af5","Type":"ContainerDied","Data":"9228aadfbb7e1badeb80b06106cac16c336a387512fcfa75fe02ea63c3ec2c4e"} Oct 31 04:12:42.340828 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:42.340716 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9228aadfbb7e1badeb80b06106cac16c336a387512fcfa75fe02ea63c3ec2c4e" Oct 31 04:12:42.965516 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:42.965471 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" containerID="cri-o://e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4" gracePeriod=45 Oct 31 04:12:43.238119 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.238089 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 04:12:43.319740 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.319700 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-tmp-dir\") pod \"5072cd0c-fc68-4271-96a5-71fd809cf75b\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " Oct 31 04:12:43.319940 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.319789 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-model-cache\") pod \"5072cd0c-fc68-4271-96a5-71fd809cf75b\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " Oct 31 04:12:43.319940 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.319890 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-dshm\") pod \"5072cd0c-fc68-4271-96a5-71fd809cf75b\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " Oct 31 04:12:43.320112 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.319962 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5072cd0c-fc68-4271-96a5-71fd809cf75b-kube-api-access-5qnz5\" (UniqueName: \"kubernetes.io/projected/5072cd0c-fc68-4271-96a5-71fd809cf75b-kube-api-access-5qnz5\") pod \"5072cd0c-fc68-4271-96a5-71fd809cf75b\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " Oct 31 04:12:43.320112 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.319965 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-tmp-dir" pod "5072cd0c-fc68-4271-96a5-71fd809cf75b" (UID: "5072cd0c-fc68-4271-96a5-71fd809cf75b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:12:43.320112 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.320019 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5072cd0c-fc68-4271-96a5-71fd809cf75b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5072cd0c-fc68-4271-96a5-71fd809cf75b-tls-certs\") pod \"5072cd0c-fc68-4271-96a5-71fd809cf75b\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " Oct 31 04:12:43.320112 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.320054 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-home\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-home\") pod \"5072cd0c-fc68-4271-96a5-71fd809cf75b\" (UID: \"5072cd0c-fc68-4271-96a5-71fd809cf75b\") " Oct 31 04:12:43.320112 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.320064 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-model-cache" pod "5072cd0c-fc68-4271-96a5-71fd809cf75b" (UID: "5072cd0c-fc68-4271-96a5-71fd809cf75b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:12:43.320472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.320360 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-home" pod "5072cd0c-fc68-4271-96a5-71fd809cf75b" (UID: "5072cd0c-fc68-4271-96a5-71fd809cf75b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:12:43.320472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.320455 4426 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-home\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:43.320580 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.320473 4426 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-tmp-dir\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:43.320580 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.320487 4426 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-model-cache\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:43.322272 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.322246 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5072cd0c-fc68-4271-96a5-71fd809cf75b-tls-certs" pod "5072cd0c-fc68-4271-96a5-71fd809cf75b" (UID: "5072cd0c-fc68-4271-96a5-71fd809cf75b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:12:43.322363 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.322287 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5072cd0c-fc68-4271-96a5-71fd809cf75b-kube-api-access-5qnz5" pod "5072cd0c-fc68-4271-96a5-71fd809cf75b" (UID: "5072cd0c-fc68-4271-96a5-71fd809cf75b"). InnerVolumeSpecName "kube-api-access-5qnz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:12:43.322363 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.322328 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-dshm" pod "5072cd0c-fc68-4271-96a5-71fd809cf75b" (UID: "5072cd0c-fc68-4271-96a5-71fd809cf75b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:12:43.351717 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.351679 4426 generic.go:356] "Generic (PLEG): container finished" podID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerID="e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4" exitCode=0 Oct 31 04:12:43.351906 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.351728 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" Oct 31 04:12:43.351906 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.351772 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" event={"ID":"5072cd0c-fc68-4271-96a5-71fd809cf75b","Type":"ContainerDied","Data":"e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4"} Oct 31 04:12:43.351906 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.351817 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt" event={"ID":"5072cd0c-fc68-4271-96a5-71fd809cf75b","Type":"ContainerDied","Data":"c4049c582982ded3e7c90eb860dd5340f334370a22421bd9082a8370cf30ed24"} Oct 31 04:12:43.351906 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.351839 4426 scope.go:122] "RemoveContainer" containerID="e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4" Oct 31 04:12:43.370712 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.370689 4426 scope.go:122] "RemoveContainer" containerID="e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4" Oct 31 04:12:43.371095 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:12:43.371060 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4\": container with ID starting with e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4 not found: ID does not exist" containerID="e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4" Oct 31 04:12:43.371350 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.371089 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4"} err="failed to get container status \"e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4\": rpc error: code = NotFound desc = could not find container \"e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4\": container with ID starting with e6aca479f86d80464ce7c40a862cc5f2aba271b995e6786c5fe8b4ad846e36e4 not found: ID does not exist" Oct 31 04:12:43.421921 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.421885 4426 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5072cd0c-fc68-4271-96a5-71fd809cf75b-dshm\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:43.422078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.421927 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qnz5\" (UniqueName: \"kubernetes.io/projected/5072cd0c-fc68-4271-96a5-71fd809cf75b-kube-api-access-5qnz5\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:43.422078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.421947 4426 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5072cd0c-fc68-4271-96a5-71fd809cf75b-tls-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:43.509008 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.508903 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt"] Oct 31 04:12:43.512156 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:43.512111 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-412ba5/distinct-2-w6-412ba5-kserve-54ff676cc-zp9mt"] Oct 31 04:12:45.128603 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:45.128561 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" path="/var/lib/kubelet/pods/5072cd0c-fc68-4271-96a5-71fd809cf75b/volumes" Oct 31 04:12:49.147408 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:49.147365 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" podUID="259d0ab8-20b4-4c18-9876-962926e794b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:12:54.144524 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:54.144486 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:12:58.047420 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:58.047385 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8"] Oct 31 04:12:58.094055 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:58.094023 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p"] Oct 31 04:12:58.097309 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:58.097284 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd58ce20-k2js8"] Oct 31 04:12:58.100224 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:58.100191 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61276fbd-l877p"] Oct 31 04:12:59.130704 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:59.130663 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab6c7b8b-f227-415c-9149-28650db67b48" path="/var/lib/kubelet/pods/ab6c7b8b-f227-415c-9149-28650db67b48/volumes" Oct 31 04:12:59.131369 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:59.131348 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c4f03e-0e39-4532-8ef1-3904dd026460" path="/var/lib/kubelet/pods/e9c4f03e-0e39-4532-8ef1-3904dd026460/volumes" Oct 31 04:12:59.148296 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:12:59.148265 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" podUID="259d0ab8-20b4-4c18-9876-962926e794b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:13:00.216004 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.215936 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 04:13:00.293390 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.293355 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259d0ab8-20b4-4c18-9876-962926e794b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259d0ab8-20b4-4c18-9876-962926e794b9-plugins-config-volume\") pod \"259d0ab8-20b4-4c18-9876-962926e794b9\" (UID: \"259d0ab8-20b4-4c18-9876-962926e794b9\") " Oct 31 04:13:00.293589 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.293450 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259d0ab8-20b4-4c18-9876-962926e794b9-kube-api-access-8qjfh\" (UniqueName: \"kubernetes.io/projected/259d0ab8-20b4-4c18-9876-962926e794b9-kube-api-access-8qjfh\") pod \"259d0ab8-20b4-4c18-9876-962926e794b9\" (UID: \"259d0ab8-20b4-4c18-9876-962926e794b9\") " Oct 31 04:13:00.293815 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.293788 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259d0ab8-20b4-4c18-9876-962926e794b9-plugins-config-volume" pod "259d0ab8-20b4-4c18-9876-962926e794b9" (UID: "259d0ab8-20b4-4c18-9876-962926e794b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:13:00.295594 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.295571 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259d0ab8-20b4-4c18-9876-962926e794b9-kube-api-access-8qjfh" pod "259d0ab8-20b4-4c18-9876-962926e794b9" (UID: "259d0ab8-20b4-4c18-9876-962926e794b9"). InnerVolumeSpecName "kube-api-access-8qjfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:13:00.394613 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.394556 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qjfh\" (UniqueName: \"kubernetes.io/projected/259d0ab8-20b4-4c18-9876-962926e794b9-kube-api-access-8qjfh\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:13:00.394613 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.394606 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259d0ab8-20b4-4c18-9876-962926e794b9-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:13:00.479626 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.479535 4426 generic.go:356] "Generic (PLEG): container finished" podID="259d0ab8-20b4-4c18-9876-962926e794b9" containerID="1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8" exitCode=0 Oct 31 04:13:00.479626 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.479579 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" Oct 31 04:13:00.479878 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.479627 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" event={"ID":"259d0ab8-20b4-4c18-9876-962926e794b9","Type":"ContainerDied","Data":"1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8"} Oct 31 04:13:00.479878 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.479687 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl" event={"ID":"259d0ab8-20b4-4c18-9876-962926e794b9","Type":"ContainerDied","Data":"ef5dd8e2c1fd6926704336b8252b537ce98edd2e9d296bac0abe89a536449647"} Oct 31 04:13:00.479878 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.479714 4426 scope.go:122] "RemoveContainer" containerID="1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8" Oct 31 04:13:00.492773 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.492755 4426 scope.go:122] "RemoveContainer" containerID="1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8" Oct 31 04:13:00.493067 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:13:00.493048 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8\": container with ID starting with 1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8 not found: ID does not exist" containerID="1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8" Oct 31 04:13:00.493164 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.493076 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8"} err="failed to get container status \"1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8\": rpc error: code = NotFound desc = could not find container \"1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8\": container with ID starting with 1a26c8b6eaf44a9f7bc7c27644e26a3c3c1deda1bafd7487e7193a1f781f4fa8 not found: ID does not exist" Oct 31 04:13:00.634229 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.634190 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl"] Oct 31 04:13:00.637043 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:00.637016 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-412ba5-596d748d8c-j2hsl"] Oct 31 04:13:01.129972 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:01.129933 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259d0ab8-20b4-4c18-9876-962926e794b9" path="/var/lib/kubelet/pods/259d0ab8-20b4-4c18-9876-962926e794b9/volumes" Oct 31 04:13:02.847383 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:13:02.847337 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:13:08.345969 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.345918 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-565fd9c896-565x2"] Oct 31 04:13:08.346534 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.346193 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-565fd9c896-565x2" podUID="2df4691e-00da-49e4-bb70-02ad5fad1625" containerName="manager" containerID="cri-o://ed99f6d7f9c9965cba8faa9b6ffd7e6e3abf7f934a34f6037c05fec82ce8394b" gracePeriod=30 Oct 31 04:13:08.539308 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.539265 4426 generic.go:356] "Generic (PLEG): container finished" podID="2df4691e-00da-49e4-bb70-02ad5fad1625" containerID="ed99f6d7f9c9965cba8faa9b6ffd7e6e3abf7f934a34f6037c05fec82ce8394b" exitCode=0 Oct 31 04:13:08.539528 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.539499 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-565x2" event={"ID":"2df4691e-00da-49e4-bb70-02ad5fad1625","Type":"ContainerDied","Data":"ed99f6d7f9c9965cba8faa9b6ffd7e6e3abf7f934a34f6037c05fec82ce8394b"} Oct 31 04:13:08.539616 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.539576 4426 scope.go:122] "RemoveContainer" containerID="22b50971fa3d01f6d36fd0d59a0d1e422c1cedaba6f9a9b4998c815328c8e5fc" Oct 31 04:13:08.616567 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.616534 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 04:13:08.670978 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.670941 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-webhook-cert\") pod \"2df4691e-00da-49e4-bb70-02ad5fad1625\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " Oct 31 04:13:08.671197 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.671153 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-metrics-certs\") pod \"2df4691e-00da-49e4-bb70-02ad5fad1625\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " Oct 31 04:13:08.671272 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.671226 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df4691e-00da-49e4-bb70-02ad5fad1625-kube-api-access-b8bl7\" (UniqueName: \"kubernetes.io/projected/2df4691e-00da-49e4-bb70-02ad5fad1625-kube-api-access-b8bl7\") pod \"2df4691e-00da-49e4-bb70-02ad5fad1625\" (UID: \"2df4691e-00da-49e4-bb70-02ad5fad1625\") " Oct 31 04:13:08.673276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.673241 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-webhook-cert" pod "2df4691e-00da-49e4-bb70-02ad5fad1625" (UID: "2df4691e-00da-49e4-bb70-02ad5fad1625"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:13:08.673735 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.673714 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-metrics-certs" pod "2df4691e-00da-49e4-bb70-02ad5fad1625" (UID: "2df4691e-00da-49e4-bb70-02ad5fad1625"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:13:08.673836 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.673728 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df4691e-00da-49e4-bb70-02ad5fad1625-kube-api-access-b8bl7" pod "2df4691e-00da-49e4-bb70-02ad5fad1625" (UID: "2df4691e-00da-49e4-bb70-02ad5fad1625"). InnerVolumeSpecName "kube-api-access-b8bl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:13:08.772650 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.772614 4426 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-metrics-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:13:08.772650 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.772646 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8bl7\" (UniqueName: \"kubernetes.io/projected/2df4691e-00da-49e4-bb70-02ad5fad1625-kube-api-access-b8bl7\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:13:08.772870 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:08.772664 4426 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2df4691e-00da-49e4-bb70-02ad5fad1625-webhook-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:13:09.547782 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:09.547747 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565fd9c896-565x2" Oct 31 04:13:09.548310 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:09.547785 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-565x2" event={"ID":"2df4691e-00da-49e4-bb70-02ad5fad1625","Type":"ContainerDied","Data":"85a7eacb3ada90e029b94b412f52226af0d2b7578bcf5eacd34f68bad3f52708"} Oct 31 04:13:09.548310 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:09.547879 4426 scope.go:122] "RemoveContainer" containerID="ed99f6d7f9c9965cba8faa9b6ffd7e6e3abf7f934a34f6037c05fec82ce8394b" Oct 31 04:13:09.702017 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:09.701978 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-565fd9c896-565x2"] Oct 31 04:13:09.748634 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:09.748592 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-565fd9c896-565x2"] Oct 31 04:13:10.623483 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.623440 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-565fd9c896-gf4gh"] Oct 31 04:13:10.624143 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624104 4426 cpu_manager.go:414] "containerMap: removing container" podUID="4d3609f5-449f-4506-ae90-2fceb42b9af5" containerName="revoke-keys" Oct 31 04:13:10.624203 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624144 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3609f5-449f-4506-ae90-2fceb42b9af5" containerName="revoke-keys" Oct 31 04:13:10.624203 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624171 4426 cpu_manager.go:414] "containerMap: removing container" podUID="2df4691e-00da-49e4-bb70-02ad5fad1625" containerName="manager" Oct 31 04:13:10.624203 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624179 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df4691e-00da-49e4-bb70-02ad5fad1625" containerName="manager" Oct 31 04:13:10.624330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624203 4426 cpu_manager.go:414] "containerMap: removing container" podUID="259d0ab8-20b4-4c18-9876-962926e794b9" containerName="payload-pre-processing" Oct 31 04:13:10.624330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624213 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259d0ab8-20b4-4c18-9876-962926e794b9" containerName="payload-pre-processing" Oct 31 04:13:10.624330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624234 4426 cpu_manager.go:414] "containerMap: removing container" podUID="2df4691e-00da-49e4-bb70-02ad5fad1625" containerName="manager" Oct 31 04:13:10.624330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624242 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df4691e-00da-49e4-bb70-02ad5fad1625" containerName="manager" Oct 31 04:13:10.624330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624258 4426 cpu_manager.go:414] "containerMap: removing container" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" Oct 31 04:13:10.624330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624266 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" Oct 31 04:13:10.624330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624278 4426 cpu_manager.go:414] "containerMap: removing container" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" Oct 31 04:13:10.624330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624285 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" Oct 31 04:13:10.624602 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624408 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3609f5-449f-4506-ae90-2fceb42b9af5" containerName="revoke-keys" Oct 31 04:13:10.624602 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624425 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df4691e-00da-49e4-bb70-02ad5fad1625" containerName="manager" Oct 31 04:13:10.624602 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624436 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="259d0ab8-20b4-4c18-9876-962926e794b9" containerName="payload-pre-processing" Oct 31 04:13:10.624602 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624446 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df4691e-00da-49e4-bb70-02ad5fad1625" containerName="manager" Oct 31 04:13:10.624602 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624456 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="5072cd0c-fc68-4271-96a5-71fd809cf75b" containerName="main" Oct 31 04:13:10.624602 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.624468 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="85150ab8-f8d1-40d2-b540-0f65db33199b" containerName="istio-proxy" Oct 31 04:13:10.629733 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.629701 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:10.678820 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.678773 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 04:13:10.692548 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.692506 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-webhook-cert\") pod \"maas-controller-565fd9c896-gf4gh\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:10.692548 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.692561 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-metrics-certs\") pod \"maas-controller-565fd9c896-gf4gh\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:10.692806 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.692667 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrkz\" (UniqueName: \"kubernetes.io/projected/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-kube-api-access-7wrkz\") pod \"maas-controller-565fd9c896-gf4gh\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:10.725248 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.725179 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 04:13:10.725248 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.725191 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8xpcq\"" Oct 31 04:13:10.725525 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.725278 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 04:13:10.729506 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.729477 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-565fd9c896-gf4gh"] Oct 31 04:13:10.793691 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.793634 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-webhook-cert\") pod \"maas-controller-565fd9c896-gf4gh\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:10.793691 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.793693 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-metrics-certs\") pod \"maas-controller-565fd9c896-gf4gh\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:10.793999 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.793819 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrkz\" (UniqueName: \"kubernetes.io/projected/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-kube-api-access-7wrkz\") pod \"maas-controller-565fd9c896-gf4gh\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:10.796519 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.796486 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-webhook-cert\") pod \"maas-controller-565fd9c896-gf4gh\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:10.796706 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.796593 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-metrics-certs\") pod \"maas-controller-565fd9c896-gf4gh\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:10.824409 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.824373 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:13:10.851640 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.851603 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrkz\" (UniqueName: \"kubernetes.io/projected/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-kube-api-access-7wrkz\") pod \"maas-controller-565fd9c896-gf4gh\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:10.943522 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:10.943480 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:11.128611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:11.128579 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df4691e-00da-49e4-bb70-02ad5fad1625" path="/var/lib/kubelet/pods/2df4691e-00da-49e4-bb70-02ad5fad1625/volumes" Oct 31 04:13:11.341867 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:11.341767 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-565fd9c896-gf4gh"] Oct 31 04:13:11.346932 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:13:11.346892 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e1ea1d_1b43_4f5e_b6ad_1763932897f2.slice/crio-83886db17725f706f219009c544e34f9e86ec9a36e89d5d6a711f0d56653d803 WatchSource:0}: Error finding container 83886db17725f706f219009c544e34f9e86ec9a36e89d5d6a711f0d56653d803: Status 404 returned error can't find the container with id 83886db17725f706f219009c544e34f9e86ec9a36e89d5d6a711f0d56653d803 Oct 31 04:13:11.348907 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:11.348879 4426 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:13:11.566603 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:11.566563 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-gf4gh" event={"ID":"46e1ea1d-1b43-4f5e-b6ad-1763932897f2","Type":"ContainerStarted","Data":"83886db17725f706f219009c544e34f9e86ec9a36e89d5d6a711f0d56653d803"} Oct 31 04:13:13.605199 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:13.605096 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-gf4gh" event={"ID":"46e1ea1d-1b43-4f5e-b6ad-1763932897f2","Type":"ContainerStarted","Data":"cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777"} Oct 31 04:13:13.605704 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:13.605263 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:13.734150 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:13.733613 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-565fd9c896-gf4gh" podStartSLOduration=2.626880185 podStartE2EDuration="3.733590897s" totalImagesPullingTime="1.106710712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:13.732939808 +0000 UTC m=+2690.733565039" watchObservedRunningTime="2026-10-31 04:13:13.733590897 +0000 UTC m=+2690.734216125" Oct 31 04:13:23.681533 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:23.681505 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:13:23.695553 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:23.695520 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:13:24.615295 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:24.615256 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:26.384445 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:26.384410 4426 scope.go:122] "RemoveContainer" containerID="e614424eefa3ade289f5a098f97749e6c3928fce0b34cdfd06ed47888d849654" Oct 31 04:13:26.394705 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:26.394683 4426 scope.go:122] "RemoveContainer" containerID="261b682ccd4de07c782bbd656f68e682181308392e1db85f6c8bdc2eedae1bb9" Oct 31 04:13:26.403837 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:26.403814 4426 scope.go:122] "RemoveContainer" containerID="aaba5cd47481409894f38f0a70c82bffc63f0e12593766b3d26b0578021c1ed5" Oct 31 04:13:26.412499 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:26.412481 4426 scope.go:122] "RemoveContainer" containerID="69a1fa85b1c0d8fb1ba0b7d5c8cf380d8627da455ba0061fc450f366d748e603" Oct 31 04:13:26.421663 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:26.421639 4426 scope.go:122] "RemoveContainer" containerID="fb60859411416a940587b051518b62d63a24d3a4c275ba3f150a8f2b2ea46412" Oct 31 04:13:32.847303 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:13:32.847259 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:13:35.594699 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:35.594662 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:13:54.207756 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.207718 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-565fd9c896-gf4gh"] Oct 31 04:13:54.208241 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.207930 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-565fd9c896-gf4gh" podUID="46e1ea1d-1b43-4f5e-b6ad-1763932897f2" containerName="manager" containerID="cri-o://cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777" gracePeriod=30 Oct 31 04:13:54.463574 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.463506 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:54.622425 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.622385 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-webhook-cert\" (UniqueName: \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-webhook-cert\") pod \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " Oct 31 04:13:54.622636 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.622486 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-kube-api-access-7wrkz\" (UniqueName: \"kubernetes.io/projected/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-kube-api-access-7wrkz\") pod \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " Oct 31 04:13:54.622636 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.622548 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-metrics-certs\") pod \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\" (UID: \"46e1ea1d-1b43-4f5e-b6ad-1763932897f2\") " Oct 31 04:13:54.624584 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.624544 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-webhook-cert" pod "46e1ea1d-1b43-4f5e-b6ad-1763932897f2" (UID: "46e1ea1d-1b43-4f5e-b6ad-1763932897f2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:13:54.624754 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.624711 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-metrics-certs" pod "46e1ea1d-1b43-4f5e-b6ad-1763932897f2" (UID: "46e1ea1d-1b43-4f5e-b6ad-1763932897f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:13:54.624822 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.624766 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-kube-api-access-7wrkz" pod "46e1ea1d-1b43-4f5e-b6ad-1763932897f2" (UID: "46e1ea1d-1b43-4f5e-b6ad-1763932897f2"). InnerVolumeSpecName "kube-api-access-7wrkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:13:54.724403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.724319 4426 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-metrics-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:13:54.724403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.724349 4426 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-webhook-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:13:54.724403 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.724361 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wrkz\" (UniqueName: \"kubernetes.io/projected/46e1ea1d-1b43-4f5e-b6ad-1763932897f2-kube-api-access-7wrkz\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:13:54.891996 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.891956 4426 generic.go:356] "Generic (PLEG): container finished" podID="46e1ea1d-1b43-4f5e-b6ad-1763932897f2" containerID="cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777" exitCode=0 Oct 31 04:13:54.892204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.892010 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565fd9c896-gf4gh" Oct 31 04:13:54.892204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.892039 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-gf4gh" event={"ID":"46e1ea1d-1b43-4f5e-b6ad-1763932897f2","Type":"ContainerDied","Data":"cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777"} Oct 31 04:13:54.892204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.892099 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-gf4gh" event={"ID":"46e1ea1d-1b43-4f5e-b6ad-1763932897f2","Type":"ContainerDied","Data":"83886db17725f706f219009c544e34f9e86ec9a36e89d5d6a711f0d56653d803"} Oct 31 04:13:54.892204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.892149 4426 scope.go:122] "RemoveContainer" containerID="cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777" Oct 31 04:13:54.903214 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.903194 4426 scope.go:122] "RemoveContainer" containerID="cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777" Oct 31 04:13:54.903505 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:13:54.903485 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777\": container with ID starting with cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777 not found: ID does not exist" containerID="cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777" Oct 31 04:13:54.903570 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:54.903516 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777"} err="failed to get container status \"cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777\": rpc error: code = NotFound desc = could not find container \"cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777\": container with ID starting with cdc2955b4f87277eb3b45f484017091b3e55127bfdf598d4d1549b2e0dcc8777 not found: ID does not exist" Oct 31 04:13:55.047011 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:55.046925 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-565fd9c896-gf4gh"] Oct 31 04:13:55.093067 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:55.093030 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-565fd9c896-gf4gh"] Oct 31 04:13:55.129666 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:13:55.129633 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e1ea1d-1b43-4f5e-b6ad-1763932897f2" path="/var/lib/kubelet/pods/46e1ea1d-1b43-4f5e-b6ad-1763932897f2/volumes" Oct 31 04:14:02.845662 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:14:02.845625 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:14:06.615012 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.614931 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-565fd9c896-fjbt8"] Oct 31 04:14:06.615508 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.615438 4426 cpu_manager.go:414] "containerMap: removing container" podUID="46e1ea1d-1b43-4f5e-b6ad-1763932897f2" containerName="manager" Oct 31 04:14:06.615508 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.615453 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e1ea1d-1b43-4f5e-b6ad-1763932897f2" containerName="manager" Oct 31 04:14:06.615587 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.615543 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e1ea1d-1b43-4f5e-b6ad-1763932897f2" containerName="manager" Oct 31 04:14:06.619984 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.619962 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:06.630120 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.630096 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhmwl\" (UniqueName: \"kubernetes.io/projected/47b46629-173e-4e33-b7a0-62236a5d7251-kube-api-access-lhmwl\") pod \"maas-controller-565fd9c896-fjbt8\" (UID: \"47b46629-173e-4e33-b7a0-62236a5d7251\") " pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:06.630271 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.630168 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b46629-173e-4e33-b7a0-62236a5d7251-metrics-certs\") pod \"maas-controller-565fd9c896-fjbt8\" (UID: \"47b46629-173e-4e33-b7a0-62236a5d7251\") " pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:06.630271 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.630227 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47b46629-173e-4e33-b7a0-62236a5d7251-webhook-cert\") pod \"maas-controller-565fd9c896-fjbt8\" (UID: \"47b46629-173e-4e33-b7a0-62236a5d7251\") " pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:06.715631 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.715597 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 04:14:06.715631 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.715597 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 04:14:06.716015 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.715662 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 04:14:06.716015 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.715701 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8xpcq\"" Oct 31 04:14:06.719068 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.719042 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-565fd9c896-fjbt8"] Oct 31 04:14:06.731680 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.731652 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhmwl\" (UniqueName: \"kubernetes.io/projected/47b46629-173e-4e33-b7a0-62236a5d7251-kube-api-access-lhmwl\") pod \"maas-controller-565fd9c896-fjbt8\" (UID: \"47b46629-173e-4e33-b7a0-62236a5d7251\") " pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:06.731834 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.731696 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b46629-173e-4e33-b7a0-62236a5d7251-metrics-certs\") pod \"maas-controller-565fd9c896-fjbt8\" (UID: \"47b46629-173e-4e33-b7a0-62236a5d7251\") " pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:06.731834 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.731716 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47b46629-173e-4e33-b7a0-62236a5d7251-webhook-cert\") pod \"maas-controller-565fd9c896-fjbt8\" (UID: \"47b46629-173e-4e33-b7a0-62236a5d7251\") " pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:06.734158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.734107 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47b46629-173e-4e33-b7a0-62236a5d7251-webhook-cert\") pod \"maas-controller-565fd9c896-fjbt8\" (UID: \"47b46629-173e-4e33-b7a0-62236a5d7251\") " pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:06.734314 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.734237 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b46629-173e-4e33-b7a0-62236a5d7251-metrics-certs\") pod \"maas-controller-565fd9c896-fjbt8\" (UID: \"47b46629-173e-4e33-b7a0-62236a5d7251\") " pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:06.789250 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.789222 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhmwl\" (UniqueName: \"kubernetes.io/projected/47b46629-173e-4e33-b7a0-62236a5d7251-kube-api-access-lhmwl\") pod \"maas-controller-565fd9c896-fjbt8\" (UID: \"47b46629-173e-4e33-b7a0-62236a5d7251\") " pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:06.933298 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:06.933259 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:07.305450 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:07.305364 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-565fd9c896-fjbt8"] Oct 31 04:14:07.310600 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:14:07.310569 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b46629_173e_4e33_b7a0_62236a5d7251.slice/crio-ee94a0655919a18e8680f8d374651c0646e08fe0163998ef17377cb36e07d51b WatchSource:0}: Error finding container ee94a0655919a18e8680f8d374651c0646e08fe0163998ef17377cb36e07d51b: Status 404 returned error can't find the container with id ee94a0655919a18e8680f8d374651c0646e08fe0163998ef17377cb36e07d51b Oct 31 04:14:07.980642 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:07.980601 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-fjbt8" event={"ID":"47b46629-173e-4e33-b7a0-62236a5d7251","Type":"ContainerStarted","Data":"ee94a0655919a18e8680f8d374651c0646e08fe0163998ef17377cb36e07d51b"} Oct 31 04:14:08.988440 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:08.988396 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565fd9c896-fjbt8" event={"ID":"47b46629-173e-4e33-b7a0-62236a5d7251","Type":"ContainerStarted","Data":"5ec8a9ed9dc0b96f0e11419436eb206f392a56a6eddb328a643f47c239ef1ff4"} Oct 31 04:14:08.988847 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:08.988474 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:09.103827 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:09.103773 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-565fd9c896-fjbt8" podStartSLOduration=2.238538195 podStartE2EDuration="3.10375778s" totalImagesPullingTime="865.219585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:09.099479377 +0000 UTC m=+2746.100104608" watchObservedRunningTime="2026-10-31 04:14:09.10375778 +0000 UTC m=+2746.104383046" Oct 31 04:14:19.997907 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:19.997868 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-565fd9c896-fjbt8" Oct 31 04:14:26.482882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:26.482844 4426 scope.go:122] "RemoveContainer" containerID="fce7a42e787f1bb09b70eb6f2271e65f3e37df6f60c94adfb7f576790a0a69e5" Oct 31 04:14:26.492791 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:26.492771 4426 scope.go:122] "RemoveContainer" containerID="3a35f3206519312e580ed5680e99caa314478e095891b49fdda6b7f16bed5530" Oct 31 04:14:26.504770 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:26.504746 4426 scope.go:122] "RemoveContainer" containerID="a91162c79ecad5b28341396ae918c16d740d330ad1fdc76c25048164c71aaacc" Oct 31 04:14:26.513688 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:26.513665 4426 scope.go:122] "RemoveContainer" containerID="c92f2ae5c57464d27655e885e0fa9222f7e9df69b9827ab5c0f7fb194f530c0f" Oct 31 04:14:32.845319 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:14:32.845277 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:14:45.057604 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:45.057569 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv"] Oct 31 04:14:45.104073 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:45.104032 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b4432d-khtvv"] Oct 31 04:14:45.130077 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:45.130038 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a456a46-63ad-4621-b430-a9980f6d5c0f" path="/var/lib/kubelet/pods/7a456a46-63ad-4621-b430-a9980f6d5c0f/volumes" Oct 31 04:14:59.817326 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:14:59.817292 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:15:02.853594 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:15:02.849060 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:15:08.532366 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.532324 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79"] Oct 31 04:15:08.536589 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.536562 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:08.565699 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.565662 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpq79\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:08.565854 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.565734 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm97t\" (UniqueName: \"kubernetes.io/projected/b96b434b-abdf-4288-be00-ad04dcf46521-kube-api-access-mm97t\") pod \"maas-api-revoke-sub-c1e271a9-kpq79\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:08.565854 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.565836 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpq79\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:08.587894 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.587861 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:15:08.634402 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.634373 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:15:08.634571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.634365 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:15:08.634571 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.634427 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:15:08.637490 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.637467 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79"] Oct 31 04:15:08.667082 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.667049 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpq79\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:08.667280 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.667144 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm97t\" (UniqueName: \"kubernetes.io/projected/b96b434b-abdf-4288-be00-ad04dcf46521-kube-api-access-mm97t\") pod \"maas-api-revoke-sub-c1e271a9-kpq79\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:08.667280 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.667197 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpq79\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:08.667936 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.667915 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpq79\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:08.669623 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.669595 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpq79\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:08.724743 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.724702 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm97t\" (UniqueName: \"kubernetes.io/projected/b96b434b-abdf-4288-be00-ad04dcf46521-kube-api-access-mm97t\") pod \"maas-api-revoke-sub-c1e271a9-kpq79\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:08.850072 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:08.849979 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:09.223558 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:09.223519 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79"] Oct 31 04:15:09.228826 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:15:09.228795 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96b434b_abdf_4288_be00_ad04dcf46521.slice/crio-c2dc476631c2628ca0782a7cd3a74fd4d2a8a9a58ec28e3f809f87e8d90eaab2 WatchSource:0}: Error finding container c2dc476631c2628ca0782a7cd3a74fd4d2a8a9a58ec28e3f809f87e8d90eaab2: Status 404 returned error can't find the container with id c2dc476631c2628ca0782a7cd3a74fd4d2a8a9a58ec28e3f809f87e8d90eaab2 Oct 31 04:15:09.431629 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:09.431595 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" event={"ID":"b96b434b-abdf-4288-be00-ad04dcf46521","Type":"ContainerStarted","Data":"c2dc476631c2628ca0782a7cd3a74fd4d2a8a9a58ec28e3f809f87e8d90eaab2"} Oct 31 04:15:10.439925 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:10.439883 4426 generic.go:356] "Generic (PLEG): container finished" podID="b96b434b-abdf-4288-be00-ad04dcf46521" containerID="9ccc428bb50eddd841ecf2cd7caeed9302b10d295eb55877647a530efac6372a" exitCode=0 Oct 31 04:15:10.440435 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:10.439965 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" event={"ID":"b96b434b-abdf-4288-be00-ad04dcf46521","Type":"ContainerDied","Data":"9ccc428bb50eddd841ecf2cd7caeed9302b10d295eb55877647a530efac6372a"} Oct 31 04:15:11.586836 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:11.586810 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:11.697220 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:11.697107 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96b434b-abdf-4288-be00-ad04dcf46521-kube-api-access-mm97t\" (UniqueName: \"kubernetes.io/projected/b96b434b-abdf-4288-be00-ad04dcf46521-kube-api-access-mm97t\") pod \"b96b434b-abdf-4288-be00-ad04dcf46521\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " Oct 31 04:15:11.697412 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:11.697244 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-openshift-ca\") pod \"b96b434b-abdf-4288-be00-ad04dcf46521\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " Oct 31 04:15:11.697412 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:11.697340 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-xks-ca\") pod \"b96b434b-abdf-4288-be00-ad04dcf46521\" (UID: \"b96b434b-abdf-4288-be00-ad04dcf46521\") " Oct 31 04:15:11.697755 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:11.697692 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-openshift-ca" pod "b96b434b-abdf-4288-be00-ad04dcf46521" (UID: "b96b434b-abdf-4288-be00-ad04dcf46521"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:15:11.697900 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:11.697860 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:15:11.700039 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:11.699991 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96b434b-abdf-4288-be00-ad04dcf46521-kube-api-access-mm97t" pod "b96b434b-abdf-4288-be00-ad04dcf46521" (UID: "b96b434b-abdf-4288-be00-ad04dcf46521"). InnerVolumeSpecName "kube-api-access-mm97t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:15:11.700670 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:11.700644 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-xks-ca" pod "b96b434b-abdf-4288-be00-ad04dcf46521" (UID: "b96b434b-abdf-4288-be00-ad04dcf46521"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:15:11.799233 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:11.799191 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm97t\" (UniqueName: \"kubernetes.io/projected/b96b434b-abdf-4288-be00-ad04dcf46521-kube-api-access-mm97t\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:15:11.799233 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:11.799227 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96b434b-abdf-4288-be00-ad04dcf46521-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:15:12.454093 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:12.454063 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" Oct 31 04:15:12.454266 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:12.454104 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79" event={"ID":"b96b434b-abdf-4288-be00-ad04dcf46521","Type":"ContainerDied","Data":"c2dc476631c2628ca0782a7cd3a74fd4d2a8a9a58ec28e3f809f87e8d90eaab2"} Oct 31 04:15:12.454266 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:12.454178 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2dc476631c2628ca0782a7cd3a74fd4d2a8a9a58ec28e3f809f87e8d90eaab2" Oct 31 04:15:21.200573 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:21.200532 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:15:26.562860 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:26.562822 4426 scope.go:122] "RemoveContainer" containerID="4473dae02e8a3cb40421b77e3de5548c6c51bd6f6c8234f0ca1c64f205c2d538" Oct 31 04:15:26.575686 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:26.575662 4426 scope.go:122] "RemoveContainer" containerID="e5021bb84e992e7ac4579630074d190b2dfc851a683e5ace44d4ac290bced67b" Oct 31 04:15:26.587878 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:26.587848 4426 scope.go:122] "RemoveContainer" containerID="46c61caf67377536b8a7cceba99447f209e9821f57c7be372de6134cec2636d7" Oct 31 04:15:26.599516 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:26.599485 4426 scope.go:122] "RemoveContainer" containerID="338ab1861b6a2160ee9319312699d311102fc6ce76b92356aa7a9bbad4e61624" Oct 31 04:15:31.050822 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:31.050783 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq"] Oct 31 04:15:31.097854 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:31.097820 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv"] Oct 31 04:15:31.100565 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:31.100537 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e87883-gqztq"] Oct 31 04:15:31.102933 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:31.102911 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a458b0ed-dmgzv"] Oct 31 04:15:31.128357 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:31.128330 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7087dcc-6049-43ce-9bfa-bd19cec2b7bd" path="/var/lib/kubelet/pods/c7087dcc-6049-43ce-9bfa-bd19cec2b7bd/volumes" Oct 31 04:15:31.128749 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:31.128735 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8" path="/var/lib/kubelet/pods/d80afc3d-ea65-4fe4-a8cb-c8342be9f6e8/volumes" Oct 31 04:15:31.796402 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:31.796352 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:15:32.846742 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:15:32.846703 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:15:52.644490 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:15:52.644450 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:16:02.846000 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:16:02.845958 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:16:09.317639 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:09.317602 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:16:26.662769 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:26.662732 4426 scope.go:122] "RemoveContainer" containerID="c20eff528353cd6d7e15a4afb9d217ad5e055b75f890396c1230d98602c30442" Oct 31 04:16:26.672503 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:26.672476 4426 scope.go:122] "RemoveContainer" containerID="4d7888889e76138a31be7ded685a61c628bd2fbe79bc3cb928a43bec820672ac" Oct 31 04:16:26.681705 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:26.681679 4426 scope.go:122] "RemoveContainer" containerID="ed614e92698f8f4cf59fc33dbb558f02b5af2cd049a60bfcace25ca96792f12a" Oct 31 04:16:30.046460 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:30.046424 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr"] Oct 31 04:16:30.093027 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:30.092977 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12a23a2-bkqxr"] Oct 31 04:16:31.045415 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:31.045382 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv"] Oct 31 04:16:31.092379 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:31.092336 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862a6eb4-x6zjv"] Oct 31 04:16:31.128461 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:31.128425 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c85669-cd8d-4766-b519-e2294dc5fc20" path="/var/lib/kubelet/pods/58c85669-cd8d-4766-b519-e2294dc5fc20/volumes" Oct 31 04:16:31.128982 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:31.128958 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8d3fac-50c1-4d10-9138-35446c73f10c" path="/var/lib/kubelet/pods/ca8d3fac-50c1-4d10-9138-35446c73f10c/volumes" Oct 31 04:16:32.845696 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:16:32.845657 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:16:40.409501 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:40.409458 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:16:51.002585 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:16:51.002548 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:17:02.847316 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:17:02.847270 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:17:03.307286 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:03.307252 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:17:25.927985 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:25.927940 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:17:26.714982 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:26.714943 4426 scope.go:122] "RemoveContainer" containerID="1845b5d24bcff5859420fdb5694bc9c5579ce00ab44119ea03bd2cf56c3b46ab" Oct 31 04:17:26.726091 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:26.726066 4426 scope.go:122] "RemoveContainer" containerID="323290b7ea403cf2e4ce5343eae5c0c546cbf27ba2c263d91c8649b5cb702e09" Oct 31 04:17:26.736998 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:26.736975 4426 scope.go:122] "RemoveContainer" containerID="2b099c10c274a0ef724e8978e1d884cf4d73fdcd878b4b28997af9be275bc190" Oct 31 04:17:26.747731 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:26.747704 4426 scope.go:122] "RemoveContainer" containerID="c9fc4774faf8b9009343eb9e6f84c9cfa72e0af421459b4c9eff01910dc381da" Oct 31 04:17:26.757323 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:26.757292 4426 scope.go:122] "RemoveContainer" containerID="54e269a0e2ec8cd41e7eddfb373b00d6ee5f4352237610913192298b9142831f" Oct 31 04:17:26.766035 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:26.766009 4426 scope.go:122] "RemoveContainer" containerID="ca0611c7d8d5579f384c8b2b9bad1c2c94f26685e46b8ddf1c1c6605349105e9" Oct 31 04:17:32.911622 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:17:32.911583 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:17:35.326196 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:35.326161 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:17:42.046989 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:42.046944 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g"] Oct 31 04:17:42.093635 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:42.093599 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7a6243-4lm9g"] Oct 31 04:17:43.128011 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:43.127974 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3609f5-449f-4506-ae90-2fceb42b9af5" path="/var/lib/kubelet/pods/4d3609f5-449f-4506-ae90-2fceb42b9af5/volumes" Oct 31 04:17:53.846837 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:17:53.846792 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:18:02.905771 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:18:02.905731 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:18:06.007487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:18:06.007445 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:18:23.731214 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:18:23.731185 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:18:23.742013 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:18:23.741986 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:18:26.234252 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:18:26.234219 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:18:26.814276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:18:26.814234 4426 scope.go:122] "RemoveContainer" containerID="d3fde62f8d5ea893a6e83511ef69be8387e5c99b7f3edbf746f0eb18cdf626ba" Oct 31 04:18:32.845688 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:18:32.845654 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:18:55.428977 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:18:55.428938 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:19:02.846514 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:19:02.846470 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:19:24.736487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:19:24.736442 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:19:26.854431 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:19:26.854379 4426 scope.go:122] "RemoveContainer" containerID="c18c09864b4becb88a5c273da7f8c3fc8d30631daf3df06ff56b9156c3493ec9" Oct 31 04:19:32.845356 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:19:32.845314 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:19:35.941414 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:19:35.941376 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:20:02.847491 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:20:02.847449 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:20:04.641976 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:20:04.641936 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:20:12.046688 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:20:12.046601 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79"] Oct 31 04:20:12.092924 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:20:12.092882 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpq79"] Oct 31 04:20:13.128342 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:20:13.128306 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96b434b-abdf-4288-be00-ad04dcf46521" path="/var/lib/kubelet/pods/b96b434b-abdf-4288-be00-ad04dcf46521/volumes" Oct 31 04:20:26.310881 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:20:26.310839 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:20:32.847275 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:20:32.847234 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:20:36.972966 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:20:36.972922 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:21:02.846680 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:21:02.846640 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:21:22.575206 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.575169 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq"] Oct 31 04:21:22.575641 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.575629 4426 cpu_manager.go:414] "containerMap: removing container" podUID="b96b434b-abdf-4288-be00-ad04dcf46521" containerName="revoke-keys" Oct 31 04:21:22.575697 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.575642 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96b434b-abdf-4288-be00-ad04dcf46521" containerName="revoke-keys" Oct 31 04:21:22.575735 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.575710 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96b434b-abdf-4288-be00-ad04dcf46521" containerName="revoke-keys" Oct 31 04:21:22.579095 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.579072 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:21:22.594454 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.594428 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d987f69-a40e-4437-830e-399d3c577a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq\" (UID: \"7d987f69-a40e-4437-830e-399d3c577a41\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:21:22.594583 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.594472 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rn4v\" (UniqueName: \"kubernetes.io/projected/7d987f69-a40e-4437-830e-399d3c577a41-kube-api-access-6rn4v\") pod \"payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq\" (UID: \"7d987f69-a40e-4437-830e-399d3c577a41\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:21:22.634056 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.634014 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq"] Oct 31 04:21:22.676995 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.676920 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f2eca2\"" Oct 31 04:21:22.676995 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.676942 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-30300582-dockercfg-bzr8x\"" Oct 31 04:21:22.695889 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.695842 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d987f69-a40e-4437-830e-399d3c577a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq\" (UID: \"7d987f69-a40e-4437-830e-399d3c577a41\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:21:22.696541 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.695924 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rn4v\" (UniqueName: \"kubernetes.io/projected/7d987f69-a40e-4437-830e-399d3c577a41-kube-api-access-6rn4v\") pod \"payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq\" (UID: \"7d987f69-a40e-4437-830e-399d3c577a41\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:21:22.696916 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.696875 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d987f69-a40e-4437-830e-399d3c577a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq\" (UID: \"7d987f69-a40e-4437-830e-399d3c577a41\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:21:22.740747 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.740705 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6"] Oct 31 04:21:22.744647 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.744621 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:21:22.758207 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.758176 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rn4v\" (UniqueName: \"kubernetes.io/projected/7d987f69-a40e-4437-830e-399d3c577a41-kube-api-access-6rn4v\") pod \"payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq\" (UID: \"7d987f69-a40e-4437-830e-399d3c577a41\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:21:22.797440 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.797394 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6\" (UID: \"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:21:22.797643 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.797449 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7t6l\" (UniqueName: \"kubernetes.io/projected/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-kube-api-access-n7t6l\") pod \"payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6\" (UID: \"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:21:22.799812 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.799781 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6"] Oct 31 04:21:22.892855 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.892820 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:21:22.898170 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.898117 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6\" (UID: \"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:21:22.898276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.898192 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7t6l\" (UniqueName: \"kubernetes.io/projected/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-kube-api-access-n7t6l\") pod \"payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6\" (UID: \"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:21:22.898830 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.898805 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6\" (UID: \"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:21:22.961963 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:22.961923 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7t6l\" (UniqueName: \"kubernetes.io/projected/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-kube-api-access-n7t6l\") pod \"payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6\" (UID: \"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:21:23.058751 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:23.058708 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:21:23.267581 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:23.267483 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq"] Oct 31 04:21:23.272399 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:21:23.272359 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d987f69_a40e_4437_830e_399d3c577a41.slice/crio-a365eb5ab6167139aa20b0b6b2648c067a946bc0a4b52b44368dda6333d33c7e WatchSource:0}: Error finding container a365eb5ab6167139aa20b0b6b2648c067a946bc0a4b52b44368dda6333d33c7e: Status 404 returned error can't find the container with id a365eb5ab6167139aa20b0b6b2648c067a946bc0a4b52b44368dda6333d33c7e Oct 31 04:21:23.431835 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:23.431796 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6"] Oct 31 04:21:23.436961 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:21:23.436932 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda736ad7b_74f2_4e7a_a4e4_b3b4d8d35cd4.slice/crio-d0a35eb5e4a00243e632a7cb1c9833bc57edbe6d95798853b9335086ef33469b WatchSource:0}: Error finding container d0a35eb5e4a00243e632a7cb1c9833bc57edbe6d95798853b9335086ef33469b: Status 404 returned error can't find the container with id d0a35eb5e4a00243e632a7cb1c9833bc57edbe6d95798853b9335086ef33469b Oct 31 04:21:24.034998 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:24.034950 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" event={"ID":"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4","Type":"ContainerStarted","Data":"39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95"} Oct 31 04:21:24.035513 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:24.035055 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" event={"ID":"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4","Type":"ContainerStarted","Data":"d0a35eb5e4a00243e632a7cb1c9833bc57edbe6d95798853b9335086ef33469b"} Oct 31 04:21:24.035513 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:24.035085 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:21:24.036816 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:24.036781 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" event={"ID":"7d987f69-a40e-4437-830e-399d3c577a41","Type":"ContainerStarted","Data":"bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa"} Oct 31 04:21:24.036952 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:24.036828 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" event={"ID":"7d987f69-a40e-4437-830e-399d3c577a41","Type":"ContainerStarted","Data":"a365eb5ab6167139aa20b0b6b2648c067a946bc0a4b52b44368dda6333d33c7e"} Oct 31 04:21:24.036952 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:24.036930 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:21:24.147426 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:24.147372 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" podStartSLOduration=2.147355846 podStartE2EDuration="2.147355846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:24.144550574 +0000 UTC m=+3181.145175810" watchObservedRunningTime="2026-10-31 04:21:24.147355846 +0000 UTC m=+3181.147981077" Oct 31 04:21:24.255869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:24.255797 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" podStartSLOduration=2.255775532 podStartE2EDuration="2.255775532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:24.25424886 +0000 UTC m=+3181.254874088" watchObservedRunningTime="2026-10-31 04:21:24.255775532 +0000 UTC m=+3181.256400762" Oct 31 04:21:26.929330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:26.929297 4426 scope.go:122] "RemoveContainer" containerID="9ccc428bb50eddd841ecf2cd7caeed9302b10d295eb55877647a530efac6372a" Oct 31 04:21:32.624499 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:32.624459 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:21:32.849456 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:21:32.849420 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:21:35.046959 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:35.046930 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:21:35.047590 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:21:35.047199 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:22:01.549073 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:01.549030 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:22:02.846651 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:22:02.846609 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:22:27.445625 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:27.445577 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:22:32.848176 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:22:32.848105 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:22:43.422500 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.422460 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln"] Oct 31 04:22:43.426387 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.426366 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:43.524788 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.524754 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:22:43.524788 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.524782 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:22:43.525009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.524756 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:22:43.525009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.524835 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:22:43.528446 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.528418 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln"] Oct 31 04:22:43.618545 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.618507 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9h58\" (UniqueName: \"kubernetes.io/projected/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-kube-api-access-t9h58\") pod \"maas-api-revoke-sub-51a8bc8a-l7tln\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:43.618745 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.618565 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a8bc8a-l7tln\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:43.618745 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.618674 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a8bc8a-l7tln\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:43.720387 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.720289 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9h58\" (UniqueName: \"kubernetes.io/projected/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-kube-api-access-t9h58\") pod \"maas-api-revoke-sub-51a8bc8a-l7tln\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:43.720387 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.720356 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a8bc8a-l7tln\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:43.720659 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.720391 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a8bc8a-l7tln\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:43.721106 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.721074 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a8bc8a-l7tln\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:43.722784 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.722762 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a8bc8a-l7tln\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:43.796783 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:43.796754 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9h58\" (UniqueName: \"kubernetes.io/projected/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-kube-api-access-t9h58\") pod \"maas-api-revoke-sub-51a8bc8a-l7tln\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:44.039944 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:44.039861 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:44.410645 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:44.410617 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln"] Oct 31 04:22:44.412370 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:22:44.412338 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eeac8df_1d2c_4446_b0c4_c54204c3a35a.slice/crio-17267c1a654e2db996646f98b9b5ef372718617e54ad2bc040694357f9b7b739 WatchSource:0}: Error finding container 17267c1a654e2db996646f98b9b5ef372718617e54ad2bc040694357f9b7b739: Status 404 returned error can't find the container with id 17267c1a654e2db996646f98b9b5ef372718617e54ad2bc040694357f9b7b739 Oct 31 04:22:44.529775 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:44.529734 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" event={"ID":"0eeac8df-1d2c-4446-b0c4-c54204c3a35a","Type":"ContainerStarted","Data":"17267c1a654e2db996646f98b9b5ef372718617e54ad2bc040694357f9b7b739"} Oct 31 04:22:45.538095 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:45.538008 4426 generic.go:356] "Generic (PLEG): container finished" podID="0eeac8df-1d2c-4446-b0c4-c54204c3a35a" containerID="aa1ee9ead03714aefe21ccadd1d7ef1f2a636c618be7dffc0789635c8d3df319" exitCode=0 Oct 31 04:22:45.538095 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:45.538083 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" event={"ID":"0eeac8df-1d2c-4446-b0c4-c54204c3a35a","Type":"ContainerDied","Data":"aa1ee9ead03714aefe21ccadd1d7ef1f2a636c618be7dffc0789635c8d3df319"} Oct 31 04:22:46.681072 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:46.681044 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:46.746406 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:46.746376 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-xks-ca\") pod \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " Oct 31 04:22:46.746594 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:46.746421 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-openshift-ca\") pod \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " Oct 31 04:22:46.746594 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:46.746453 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-kube-api-access-t9h58\" (UniqueName: \"kubernetes.io/projected/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-kube-api-access-t9h58\") pod \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\" (UID: \"0eeac8df-1d2c-4446-b0c4-c54204c3a35a\") " Oct 31 04:22:46.746854 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:46.746829 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-openshift-ca" pod "0eeac8df-1d2c-4446-b0c4-c54204c3a35a" (UID: "0eeac8df-1d2c-4446-b0c4-c54204c3a35a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:46.748502 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:46.748464 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-xks-ca" pod "0eeac8df-1d2c-4446-b0c4-c54204c3a35a" (UID: "0eeac8df-1d2c-4446-b0c4-c54204c3a35a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:22:46.748626 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:46.748591 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-kube-api-access-t9h58" pod "0eeac8df-1d2c-4446-b0c4-c54204c3a35a" (UID: "0eeac8df-1d2c-4446-b0c4-c54204c3a35a"). InnerVolumeSpecName "kube-api-access-t9h58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:46.847530 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:46.847447 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:22:46.847530 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:46.847480 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:22:46.847530 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:46.847492 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9h58\" (UniqueName: \"kubernetes.io/projected/0eeac8df-1d2c-4446-b0c4-c54204c3a35a-kube-api-access-t9h58\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:22:47.552554 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:47.552526 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" Oct 31 04:22:47.552742 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:47.552520 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln" event={"ID":"0eeac8df-1d2c-4446-b0c4-c54204c3a35a","Type":"ContainerDied","Data":"17267c1a654e2db996646f98b9b5ef372718617e54ad2bc040694357f9b7b739"} Oct 31 04:22:47.552742 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:47.552657 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17267c1a654e2db996646f98b9b5ef372718617e54ad2bc040694357f9b7b739" Oct 31 04:22:55.620260 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.620224 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p"] Oct 31 04:22:55.620958 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.620936 4426 cpu_manager.go:414] "containerMap: removing container" podUID="0eeac8df-1d2c-4446-b0c4-c54204c3a35a" containerName="revoke-keys" Oct 31 04:22:55.621064 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.620960 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eeac8df-1d2c-4446-b0c4-c54204c3a35a" containerName="revoke-keys" Oct 31 04:22:55.621160 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.621146 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eeac8df-1d2c-4446-b0c4-c54204c3a35a" containerName="revoke-keys" Oct 31 04:22:55.624582 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.624561 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:55.678625 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.678596 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p"] Oct 31 04:22:55.722206 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.722173 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:22:55.722386 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.722218 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:22:55.722812 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.722786 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:22:55.722912 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.722890 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:22:55.733581 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.733548 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrw9x\" (UniqueName: \"kubernetes.io/projected/9694a383-282b-4b68-9491-3e7eb3340219-kube-api-access-qrw9x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:55.733684 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.733607 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:55.733731 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.733699 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:55.834487 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.834442 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:55.834709 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.834507 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:55.834709 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.834608 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrw9x\" (UniqueName: \"kubernetes.io/projected/9694a383-282b-4b68-9491-3e7eb3340219-kube-api-access-qrw9x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:55.835495 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.835464 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:55.837212 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.837189 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:55.892164 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.892137 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrw9x\" (UniqueName: \"kubernetes.io/projected/9694a383-282b-4b68-9491-3e7eb3340219-kube-api-access-qrw9x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:55.940576 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:55.940538 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:56.313950 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:56.313920 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p"] Oct 31 04:22:56.315418 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:22:56.315391 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9694a383_282b_4b68_9491_3e7eb3340219.slice/crio-01cfaf469b2d8c0fff9c6df56cc3535ff42a84c2c3320eb3e5235437555eb54a WatchSource:0}: Error finding container 01cfaf469b2d8c0fff9c6df56cc3535ff42a84c2c3320eb3e5235437555eb54a: Status 404 returned error can't find the container with id 01cfaf469b2d8c0fff9c6df56cc3535ff42a84c2c3320eb3e5235437555eb54a Oct 31 04:22:56.617645 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:56.617609 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" event={"ID":"9694a383-282b-4b68-9491-3e7eb3340219","Type":"ContainerStarted","Data":"01cfaf469b2d8c0fff9c6df56cc3535ff42a84c2c3320eb3e5235437555eb54a"} Oct 31 04:22:57.627253 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:57.627205 4426 generic.go:356] "Generic (PLEG): container finished" podID="9694a383-282b-4b68-9491-3e7eb3340219" containerID="cd70ac86812dce812ad63e7128c2b4064dc4540a6989bf4c4fabf8d933702d5d" exitCode=0 Oct 31 04:22:57.627696 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:57.627293 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" event={"ID":"9694a383-282b-4b68-9491-3e7eb3340219","Type":"ContainerDied","Data":"cd70ac86812dce812ad63e7128c2b4064dc4540a6989bf4c4fabf8d933702d5d"} Oct 31 04:22:58.240695 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:58.240660 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:22:58.787090 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:58.787061 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:58.869146 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:58.869088 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-xks-ca\") pod \"9694a383-282b-4b68-9491-3e7eb3340219\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " Oct 31 04:22:58.869322 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:58.869156 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9694a383-282b-4b68-9491-3e7eb3340219-kube-api-access-qrw9x\" (UniqueName: \"kubernetes.io/projected/9694a383-282b-4b68-9491-3e7eb3340219-kube-api-access-qrw9x\") pod \"9694a383-282b-4b68-9491-3e7eb3340219\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " Oct 31 04:22:58.869322 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:58.869265 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-openshift-ca\") pod \"9694a383-282b-4b68-9491-3e7eb3340219\" (UID: \"9694a383-282b-4b68-9491-3e7eb3340219\") " Oct 31 04:22:58.869632 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:58.869609 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-openshift-ca" pod "9694a383-282b-4b68-9491-3e7eb3340219" (UID: "9694a383-282b-4b68-9491-3e7eb3340219"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:58.871478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:58.871445 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-xks-ca" pod "9694a383-282b-4b68-9491-3e7eb3340219" (UID: "9694a383-282b-4b68-9491-3e7eb3340219"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:22:58.871478 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:58.871459 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9694a383-282b-4b68-9491-3e7eb3340219-kube-api-access-qrw9x" pod "9694a383-282b-4b68-9491-3e7eb3340219" (UID: "9694a383-282b-4b68-9491-3e7eb3340219"). InnerVolumeSpecName "kube-api-access-qrw9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:58.970182 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:58.970143 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:22:58.970182 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:58.970175 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrw9x\" (UniqueName: \"kubernetes.io/projected/9694a383-282b-4b68-9491-3e7eb3340219-kube-api-access-qrw9x\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:22:58.970182 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:58.970186 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9694a383-282b-4b68-9491-3e7eb3340219-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:22:59.639530 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:59.639499 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" Oct 31 04:22:59.639530 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:59.639523 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p" event={"ID":"9694a383-282b-4b68-9491-3e7eb3340219","Type":"ContainerDied","Data":"01cfaf469b2d8c0fff9c6df56cc3535ff42a84c2c3320eb3e5235437555eb54a"} Oct 31 04:22:59.639811 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:22:59.639574 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01cfaf469b2d8c0fff9c6df56cc3535ff42a84c2c3320eb3e5235437555eb54a" Oct 31 04:23:02.847171 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:23:02.847114 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:23:07.016350 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:07.016268 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6"] Oct 31 04:23:07.114551 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:07.114513 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq"] Oct 31 04:23:12.016785 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:12.016714 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" podUID="a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4" containerName="payload-processing" containerID="cri-o://39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95" gracePeriod=25 Oct 31 04:23:12.115866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:12.115819 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" podUID="7d987f69-a40e-4437-830e-399d3c577a41" containerName="payload-pre-processing" containerID="cri-o://bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa" gracePeriod=25 Oct 31 04:23:15.045264 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:15.045196 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" podUID="7d987f69-a40e-4437-830e-399d3c577a41" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:23:15.045903 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:15.045212 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" podUID="a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:23:16.567589 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:16.567545 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p"] Oct 31 04:23:16.613327 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:16.613295 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f2eca2-n9f4p"] Oct 31 04:23:17.128789 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:17.128759 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9694a383-282b-4b68-9491-3e7eb3340219" path="/var/lib/kubelet/pods/9694a383-282b-4b68-9491-3e7eb3340219/volumes" Oct 31 04:23:19.014947 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:19.014910 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:23:23.789607 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:23.789576 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:23:23.796523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:23.796498 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:23:25.045396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:25.045351 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" podUID="a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:23:25.045766 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:25.045565 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" podUID="7d987f69-a40e-4437-830e-399d3c577a41" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:23:32.277325 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.277253 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:23:32.296717 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.296681 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-kube-api-access-n7t6l\" (UniqueName: \"kubernetes.io/projected/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-kube-api-access-n7t6l\") pod \"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4\" (UID: \"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4\") " Oct 31 04:23:32.296884 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.296780 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-plugins-config-volume\") pod \"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4\" (UID: \"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4\") " Oct 31 04:23:32.297576 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.297548 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-plugins-config-volume" pod "a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4" (UID: "a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:32.300327 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.300296 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-kube-api-access-n7t6l" pod "a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4" (UID: "a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4"). InnerVolumeSpecName "kube-api-access-n7t6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:32.358810 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.358784 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:23:32.398223 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.398190 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d987f69-a40e-4437-830e-399d3c577a41-kube-api-access-6rn4v\" (UniqueName: \"kubernetes.io/projected/7d987f69-a40e-4437-830e-399d3c577a41-kube-api-access-6rn4v\") pod \"7d987f69-a40e-4437-830e-399d3c577a41\" (UID: \"7d987f69-a40e-4437-830e-399d3c577a41\") " Oct 31 04:23:32.398410 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.398303 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d987f69-a40e-4437-830e-399d3c577a41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d987f69-a40e-4437-830e-399d3c577a41-plugins-config-volume\") pod \"7d987f69-a40e-4437-830e-399d3c577a41\" (UID: \"7d987f69-a40e-4437-830e-399d3c577a41\") " Oct 31 04:23:32.398649 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.398632 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7t6l\" (UniqueName: \"kubernetes.io/projected/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-kube-api-access-n7t6l\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:23:32.398732 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.398654 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:23:32.398732 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.398652 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d987f69-a40e-4437-830e-399d3c577a41-plugins-config-volume" pod "7d987f69-a40e-4437-830e-399d3c577a41" (UID: "7d987f69-a40e-4437-830e-399d3c577a41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:32.400523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.400494 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d987f69-a40e-4437-830e-399d3c577a41-kube-api-access-6rn4v" pod "7d987f69-a40e-4437-830e-399d3c577a41" (UID: "7d987f69-a40e-4437-830e-399d3c577a41"). InnerVolumeSpecName "kube-api-access-6rn4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:32.499896 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.499855 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rn4v\" (UniqueName: \"kubernetes.io/projected/7d987f69-a40e-4437-830e-399d3c577a41-kube-api-access-6rn4v\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:23:32.499896 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.499896 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d987f69-a40e-4437-830e-399d3c577a41-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:23:32.839523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.839485 4426 generic.go:356] "Generic (PLEG): container finished" podID="a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4" containerID="39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95" exitCode=0 Oct 31 04:23:32.839737 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.839533 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" Oct 31 04:23:32.839737 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.839572 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" event={"ID":"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4","Type":"ContainerDied","Data":"39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95"} Oct 31 04:23:32.839737 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.839639 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6" event={"ID":"a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4","Type":"ContainerDied","Data":"d0a35eb5e4a00243e632a7cb1c9833bc57edbe6d95798853b9335086ef33469b"} Oct 31 04:23:32.839737 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.839669 4426 scope.go:122] "RemoveContainer" containerID="39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95" Oct 31 04:23:32.841628 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.841603 4426 generic.go:356] "Generic (PLEG): container finished" podID="7d987f69-a40e-4437-830e-399d3c577a41" containerID="bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa" exitCode=0 Oct 31 04:23:32.841748 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.841651 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" Oct 31 04:23:32.841748 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.841727 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" event={"ID":"7d987f69-a40e-4437-830e-399d3c577a41","Type":"ContainerDied","Data":"bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa"} Oct 31 04:23:32.841844 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.841784 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq" event={"ID":"7d987f69-a40e-4437-830e-399d3c577a41","Type":"ContainerDied","Data":"a365eb5ab6167139aa20b0b6b2648c067a946bc0a4b52b44368dda6333d33c7e"} Oct 31 04:23:32.848041 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:23:32.848001 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:23:32.855640 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.855608 4426 scope.go:122] "RemoveContainer" containerID="39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95" Oct 31 04:23:32.856227 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:23:32.856094 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95\": container with ID starting with 39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95 not found: ID does not exist" containerID="39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95" Oct 31 04:23:32.856227 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.856179 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95"} err="failed to get container status \"39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95\": rpc error: code = NotFound desc = could not find container \"39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95\": container with ID starting with 39aff2c0217e841f978ec5a8d9c4044fa6343f303c29fe43d93e4386a8ccfa95 not found: ID does not exist" Oct 31 04:23:32.856227 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.856204 4426 scope.go:122] "RemoveContainer" containerID="bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa" Oct 31 04:23:32.866222 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.866198 4426 scope.go:122] "RemoveContainer" containerID="bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa" Oct 31 04:23:32.866541 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:23:32.866519 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa\": container with ID starting with bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa not found: ID does not exist" containerID="bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa" Oct 31 04:23:32.866583 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.866551 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa"} err="failed to get container status \"bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa\": rpc error: code = NotFound desc = could not find container \"bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa\": container with ID starting with bcc250e87c3f0a46ccbf5df01b161e603262b5b4ce3bcbfa9a5af8d46b275efa not found: ID does not exist" Oct 31 04:23:32.993650 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.993611 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq"] Oct 31 04:23:32.999330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:32.999299 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f2eca2-76fb6nz9wq"] Oct 31 04:23:33.104578 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:33.104490 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6"] Oct 31 04:23:33.107137 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:33.107100 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f2eca2-8fc78d465znsf6"] Oct 31 04:23:33.129802 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:33.129770 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d987f69-a40e-4437-830e-399d3c577a41" path="/var/lib/kubelet/pods/7d987f69-a40e-4437-830e-399d3c577a41/volumes" Oct 31 04:23:33.130204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:33.130188 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4" path="/var/lib/kubelet/pods/a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4/volumes" Oct 31 04:23:44.310728 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.310691 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m"] Oct 31 04:23:44.311186 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.310910 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" podUID="63813f18-28ec-4dab-bd59-17d1a825c4a8" containerName="manager" containerID="cri-o://0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6" gracePeriod=10 Oct 31 04:23:44.574583 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.574518 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 04:23:44.622893 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.622857 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63813f18-28ec-4dab-bd59-17d1a825c4a8-kube-api-access-kkfp8\" (UniqueName: \"kubernetes.io/projected/63813f18-28ec-4dab-bd59-17d1a825c4a8-kube-api-access-kkfp8\") pod \"63813f18-28ec-4dab-bd59-17d1a825c4a8\" (UID: \"63813f18-28ec-4dab-bd59-17d1a825c4a8\") " Oct 31 04:23:44.623102 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.622989 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63813f18-28ec-4dab-bd59-17d1a825c4a8-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63813f18-28ec-4dab-bd59-17d1a825c4a8-extensions-socket-volume\") pod \"63813f18-28ec-4dab-bd59-17d1a825c4a8\" (UID: \"63813f18-28ec-4dab-bd59-17d1a825c4a8\") " Oct 31 04:23:44.623500 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.623474 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63813f18-28ec-4dab-bd59-17d1a825c4a8-extensions-socket-volume" pod "63813f18-28ec-4dab-bd59-17d1a825c4a8" (UID: "63813f18-28ec-4dab-bd59-17d1a825c4a8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:44.625146 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.625097 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63813f18-28ec-4dab-bd59-17d1a825c4a8-kube-api-access-kkfp8" pod "63813f18-28ec-4dab-bd59-17d1a825c4a8" (UID: "63813f18-28ec-4dab-bd59-17d1a825c4a8"). InnerVolumeSpecName "kube-api-access-kkfp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:44.724642 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.724601 4426 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63813f18-28ec-4dab-bd59-17d1a825c4a8-extensions-socket-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:23:44.724642 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.724646 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkfp8\" (UniqueName: \"kubernetes.io/projected/63813f18-28ec-4dab-bd59-17d1a825c4a8-kube-api-access-kkfp8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:23:44.920656 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.920624 4426 generic.go:356] "Generic (PLEG): container finished" podID="63813f18-28ec-4dab-bd59-17d1a825c4a8" containerID="0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6" exitCode=0 Oct 31 04:23:44.920865 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.920675 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" Oct 31 04:23:44.920865 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.920707 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" event={"ID":"63813f18-28ec-4dab-bd59-17d1a825c4a8","Type":"ContainerDied","Data":"0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6"} Oct 31 04:23:44.920865 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.920773 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m" event={"ID":"63813f18-28ec-4dab-bd59-17d1a825c4a8","Type":"ContainerDied","Data":"d9f60b22eb221f4c1a5ab6f5e1b0983354773401e33966160fc595a5f6eaad5e"} Oct 31 04:23:44.920865 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.920802 4426 scope.go:122] "RemoveContainer" containerID="0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6" Oct 31 04:23:44.933965 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.933946 4426 scope.go:122] "RemoveContainer" containerID="0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6" Oct 31 04:23:44.934276 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:23:44.934256 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6\": container with ID starting with 0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6 not found: ID does not exist" containerID="0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6" Oct 31 04:23:44.934330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:44.934286 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6"} err="failed to get container status \"0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6\": rpc error: code = NotFound desc = could not find container \"0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6\": container with ID starting with 0e3b6bce6c9ed3774abf19323b3d59e88e3eb2693178a53d3b83c0c13f29b5c6 not found: ID does not exist" Oct 31 04:23:45.091119 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:45.091082 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m"] Oct 31 04:23:45.098442 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:45.098418 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4zz6m"] Oct 31 04:23:45.128220 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:23:45.128181 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63813f18-28ec-4dab-bd59-17d1a825c4a8" path="/var/lib/kubelet/pods/63813f18-28ec-4dab-bd59-17d1a825c4a8/volumes" Oct 31 04:24:02.845729 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:24:02.845683 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:24:32.846121 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:24:32.846082 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:24:55.771305 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771264 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt"] Oct 31 04:24:55.771737 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771723 4426 cpu_manager.go:414] "containerMap: removing container" podUID="7d987f69-a40e-4437-830e-399d3c577a41" containerName="payload-pre-processing" Oct 31 04:24:55.771783 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771737 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d987f69-a40e-4437-830e-399d3c577a41" containerName="payload-pre-processing" Oct 31 04:24:55.771783 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771751 4426 cpu_manager.go:414] "containerMap: removing container" podUID="9694a383-282b-4b68-9491-3e7eb3340219" containerName="revoke-keys" Oct 31 04:24:55.771783 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771757 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9694a383-282b-4b68-9491-3e7eb3340219" containerName="revoke-keys" Oct 31 04:24:55.771783 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771772 4426 cpu_manager.go:414] "containerMap: removing container" podUID="63813f18-28ec-4dab-bd59-17d1a825c4a8" containerName="manager" Oct 31 04:24:55.771783 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771776 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63813f18-28ec-4dab-bd59-17d1a825c4a8" containerName="manager" Oct 31 04:24:55.771783 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771782 4426 cpu_manager.go:414] "containerMap: removing container" podUID="a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4" containerName="payload-processing" Oct 31 04:24:55.771986 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771787 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4" containerName="payload-processing" Oct 31 04:24:55.771986 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771844 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="63813f18-28ec-4dab-bd59-17d1a825c4a8" containerName="manager" Oct 31 04:24:55.771986 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771851 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="a736ad7b-74f2-4e7a-a4e4-b3b4d8d35cd4" containerName="payload-processing" Oct 31 04:24:55.771986 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771861 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d987f69-a40e-4437-830e-399d3c577a41" containerName="payload-pre-processing" Oct 31 04:24:55.771986 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.771868 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="9694a383-282b-4b68-9491-3e7eb3340219" containerName="revoke-keys" Oct 31 04:24:55.775083 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.775065 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" Oct 31 04:24:55.820037 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.820002 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csnht\" (UniqueName: \"kubernetes.io/projected/471df0d0-0980-416c-a410-dabeaeaa01ee-kube-api-access-csnht\") pod \"kuadrant-operator-controller-manager-678d87766f-7pknt\" (UID: \"471df0d0-0980-416c-a410-dabeaeaa01ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" Oct 31 04:24:55.820239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.820050 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/471df0d0-0980-416c-a410-dabeaeaa01ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7pknt\" (UID: \"471df0d0-0980-416c-a410-dabeaeaa01ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" Oct 31 04:24:55.826699 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.826672 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt"] Oct 31 04:24:55.870226 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.870180 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tpf45\"" Oct 31 04:24:55.921246 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.921212 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csnht\" (UniqueName: \"kubernetes.io/projected/471df0d0-0980-416c-a410-dabeaeaa01ee-kube-api-access-csnht\") pod \"kuadrant-operator-controller-manager-678d87766f-7pknt\" (UID: \"471df0d0-0980-416c-a410-dabeaeaa01ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" Oct 31 04:24:55.921461 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.921255 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/471df0d0-0980-416c-a410-dabeaeaa01ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7pknt\" (UID: \"471df0d0-0980-416c-a410-dabeaeaa01ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" Oct 31 04:24:55.921655 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:55.921636 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/471df0d0-0980-416c-a410-dabeaeaa01ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7pknt\" (UID: \"471df0d0-0980-416c-a410-dabeaeaa01ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" Oct 31 04:24:56.006598 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:56.006564 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csnht\" (UniqueName: \"kubernetes.io/projected/471df0d0-0980-416c-a410-dabeaeaa01ee-kube-api-access-csnht\") pod \"kuadrant-operator-controller-manager-678d87766f-7pknt\" (UID: \"471df0d0-0980-416c-a410-dabeaeaa01ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" Oct 31 04:24:56.087707 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:56.087602 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" Oct 31 04:24:56.553115 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:56.553075 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt"] Oct 31 04:24:56.554398 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:24:56.554369 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471df0d0_0980_416c_a410_dabeaeaa01ee.slice/crio-a56988c8a41a2bf8f151aeb6d5b9c00f79e2411852c033f96c14ebd102faef02 WatchSource:0}: Error finding container a56988c8a41a2bf8f151aeb6d5b9c00f79e2411852c033f96c14ebd102faef02: Status 404 returned error can't find the container with id a56988c8a41a2bf8f151aeb6d5b9c00f79e2411852c033f96c14ebd102faef02 Oct 31 04:24:57.335810 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:57.335774 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" event={"ID":"471df0d0-0980-416c-a410-dabeaeaa01ee","Type":"ContainerStarted","Data":"3bbb0346934a713fa38bb5703a3f07b15c520c660642c2797d2c251aa4f8804b"} Oct 31 04:24:57.336264 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:57.335825 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" event={"ID":"471df0d0-0980-416c-a410-dabeaeaa01ee","Type":"ContainerStarted","Data":"a56988c8a41a2bf8f151aeb6d5b9c00f79e2411852c033f96c14ebd102faef02"} Oct 31 04:24:57.336264 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:57.335909 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" Oct 31 04:24:57.544447 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:24:57.544389 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" podStartSLOduration=2.544373131 podStartE2EDuration="2.544373131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:57.541506165 +0000 UTC m=+3394.542131394" watchObservedRunningTime="2026-10-31 04:24:57.544373131 +0000 UTC m=+3394.544998360" Oct 31 04:25:02.845580 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:25:02.845536 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:25:08.345163 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:25:08.345108 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pknt" Oct 31 04:25:17.231625 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:25:17.231573 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:25:32.846072 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:25:32.846032 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:25:34.417365 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:25:34.417319 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:25:43.638388 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:25:43.638346 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:26:02.849200 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:26:02.849152 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:26:03.429794 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:03.429759 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:26:21.237021 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.236985 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq"] Oct 31 04:26:21.240544 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.240524 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:26:21.292538 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.292486 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b097c5\"" Oct 31 04:26:21.310986 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.310949 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5n8\" (UniqueName: \"kubernetes.io/projected/08afaa1d-d616-44d0-a483-8345c229bed4-kube-api-access-6z5n8\") pod \"payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq\" (UID: \"08afaa1d-d616-44d0-a483-8345c229bed4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:26:21.311197 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.311010 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08afaa1d-d616-44d0-a483-8345c229bed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq\" (UID: \"08afaa1d-d616-44d0-a483-8345c229bed4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:26:21.338744 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.338708 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-dcc892c3-dockercfg-4j29d\"" Oct 31 04:26:21.343532 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.343493 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq"] Oct 31 04:26:21.398026 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.397988 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms"] Oct 31 04:26:21.401723 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.401707 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:26:21.411756 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.411730 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5n8\" (UniqueName: \"kubernetes.io/projected/08afaa1d-d616-44d0-a483-8345c229bed4-kube-api-access-6z5n8\") pod \"payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq\" (UID: \"08afaa1d-d616-44d0-a483-8345c229bed4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:26:21.411889 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.411776 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nt7v\" (UniqueName: \"kubernetes.io/projected/9d1dacd5-6785-4091-a468-5cfcc1de41e2-kube-api-access-6nt7v\") pod \"payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms\" (UID: \"9d1dacd5-6785-4091-a468-5cfcc1de41e2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:26:21.411889 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.411805 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08afaa1d-d616-44d0-a483-8345c229bed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq\" (UID: \"08afaa1d-d616-44d0-a483-8345c229bed4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:26:21.411889 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.411820 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1dacd5-6785-4091-a468-5cfcc1de41e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms\" (UID: \"9d1dacd5-6785-4091-a468-5cfcc1de41e2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:26:21.413348 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.413316 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08afaa1d-d616-44d0-a483-8345c229bed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq\" (UID: \"08afaa1d-d616-44d0-a483-8345c229bed4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:26:21.457882 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.457846 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms"] Oct 31 04:26:21.469869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.469834 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5n8\" (UniqueName: \"kubernetes.io/projected/08afaa1d-d616-44d0-a483-8345c229bed4-kube-api-access-6z5n8\") pod \"payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq\" (UID: \"08afaa1d-d616-44d0-a483-8345c229bed4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:26:21.512587 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.512495 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1dacd5-6785-4091-a468-5cfcc1de41e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms\" (UID: \"9d1dacd5-6785-4091-a468-5cfcc1de41e2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:26:21.512777 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.512613 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nt7v\" (UniqueName: \"kubernetes.io/projected/9d1dacd5-6785-4091-a468-5cfcc1de41e2-kube-api-access-6nt7v\") pod \"payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms\" (UID: \"9d1dacd5-6785-4091-a468-5cfcc1de41e2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:26:21.513339 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.513313 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1dacd5-6785-4091-a468-5cfcc1de41e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms\" (UID: \"9d1dacd5-6785-4091-a468-5cfcc1de41e2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:26:21.554217 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.554179 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:26:21.570557 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.570529 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nt7v\" (UniqueName: \"kubernetes.io/projected/9d1dacd5-6785-4091-a468-5cfcc1de41e2-kube-api-access-6nt7v\") pod \"payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms\" (UID: \"9d1dacd5-6785-4091-a468-5cfcc1de41e2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:26:21.722277 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.722232 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:26:21.924316 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:21.924287 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq"] Oct 31 04:26:21.925203 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:26:21.925180 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08afaa1d_d616_44d0_a483_8345c229bed4.slice/crio-56fa3e963be20ad60e9f567ba42437f897a0752cb76d2bab5abe30a34bee0cae WatchSource:0}: Error finding container 56fa3e963be20ad60e9f567ba42437f897a0752cb76d2bab5abe30a34bee0cae: Status 404 returned error can't find the container with id 56fa3e963be20ad60e9f567ba42437f897a0752cb76d2bab5abe30a34bee0cae Oct 31 04:26:22.092090 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:22.092058 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms"] Oct 31 04:26:22.092988 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:26:22.092961 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1dacd5_6785_4091_a468_5cfcc1de41e2.slice/crio-5842f35e64ab4fde90fe9a1c8f25452e0b359978838958ee3469526753887169 WatchSource:0}: Error finding container 5842f35e64ab4fde90fe9a1c8f25452e0b359978838958ee3469526753887169: Status 404 returned error can't find the container with id 5842f35e64ab4fde90fe9a1c8f25452e0b359978838958ee3469526753887169 Oct 31 04:26:22.885176 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:22.885139 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" event={"ID":"9d1dacd5-6785-4091-a468-5cfcc1de41e2","Type":"ContainerStarted","Data":"9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810"} Oct 31 04:26:22.885677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:22.885203 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" event={"ID":"9d1dacd5-6785-4091-a468-5cfcc1de41e2","Type":"ContainerStarted","Data":"5842f35e64ab4fde90fe9a1c8f25452e0b359978838958ee3469526753887169"} Oct 31 04:26:22.885677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:22.885363 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:26:22.886878 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:22.886855 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" event={"ID":"08afaa1d-d616-44d0-a483-8345c229bed4","Type":"ContainerStarted","Data":"a4d29b981012b706aab681c55f28ae9f05477bfca17e121fabf4648232aa3b61"} Oct 31 04:26:22.887018 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:22.886894 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" event={"ID":"08afaa1d-d616-44d0-a483-8345c229bed4","Type":"ContainerStarted","Data":"56fa3e963be20ad60e9f567ba42437f897a0752cb76d2bab5abe30a34bee0cae"} Oct 31 04:26:22.887018 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:22.886937 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:26:22.996609 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:22.996540 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" podStartSLOduration=1.996519302 podStartE2EDuration="1.996519302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:22.994222462 +0000 UTC m=+3479.994847690" watchObservedRunningTime="2026-10-31 04:26:22.996519302 +0000 UTC m=+3479.997144531" Oct 31 04:26:23.106273 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:23.106218 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" podStartSLOduration=2.106203342 podStartE2EDuration="2.106203342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:23.103976064 +0000 UTC m=+3480.104601292" watchObservedRunningTime="2026-10-31 04:26:23.106203342 +0000 UTC m=+3480.106828570" Oct 31 04:26:31.132948 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:31.132902 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:26:32.846844 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:26:32.846806 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:26:33.896064 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:33.896034 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:26:33.896625 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:33.896114 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:26:50.210810 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:26:50.210770 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:27:02.845741 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:27:02.845702 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:27:32.846259 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:27:32.846212 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:27:36.231960 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:36.231865 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:27:47.048491 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:47.048455 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln"] Oct 31 04:27:47.094492 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:47.094444 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a8bc8a-l7tln"] Oct 31 04:27:47.128971 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:47.128943 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eeac8df-1d2c-4446-b0c4-c54204c3a35a" path="/var/lib/kubelet/pods/0eeac8df-1d2c-4446-b0c4-c54204c3a35a/volumes" Oct 31 04:27:51.393954 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.393910 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d"] Oct 31 04:27:51.397961 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.397940 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:51.453568 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.453532 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d"] Oct 31 04:27:51.496061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.496024 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:27:51.496061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.496024 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:27:51.496061 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.496039 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:27:51.496329 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.496138 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:27:51.582631 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.582588 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acd39fd9-q279d\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:51.582819 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.582737 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72j6\" (UniqueName: \"kubernetes.io/projected/abc4870f-03e0-4c54-9d91-76412b43b05b-kube-api-access-c72j6\") pod \"maas-api-revoke-sub-acd39fd9-q279d\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:51.582874 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.582830 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acd39fd9-q279d\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:51.683822 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.683728 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72j6\" (UniqueName: \"kubernetes.io/projected/abc4870f-03e0-4c54-9d91-76412b43b05b-kube-api-access-c72j6\") pod \"maas-api-revoke-sub-acd39fd9-q279d\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:51.683822 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.683789 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acd39fd9-q279d\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:51.684079 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.683858 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acd39fd9-q279d\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:51.684587 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.684562 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acd39fd9-q279d\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:51.686449 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.686421 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acd39fd9-q279d\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:51.741753 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:51.741710 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72j6\" (UniqueName: \"kubernetes.io/projected/abc4870f-03e0-4c54-9d91-76412b43b05b-kube-api-access-c72j6\") pod \"maas-api-revoke-sub-acd39fd9-q279d\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:52.011848 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:52.011762 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:52.383983 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:52.383953 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d"] Oct 31 04:27:52.384801 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:27:52.384776 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc4870f_03e0_4c54_9d91_76412b43b05b.slice/crio-adc880bcc7af7c1821234093f329dcdb025a99f58d4a733a278b959ff09fc652 WatchSource:0}: Error finding container adc880bcc7af7c1821234093f329dcdb025a99f58d4a733a278b959ff09fc652: Status 404 returned error can't find the container with id adc880bcc7af7c1821234093f329dcdb025a99f58d4a733a278b959ff09fc652 Oct 31 04:27:52.434809 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:52.434777 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" event={"ID":"abc4870f-03e0-4c54-9d91-76412b43b05b","Type":"ContainerStarted","Data":"adc880bcc7af7c1821234093f329dcdb025a99f58d4a733a278b959ff09fc652"} Oct 31 04:27:53.443780 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:53.443736 4426 generic.go:356] "Generic (PLEG): container finished" podID="abc4870f-03e0-4c54-9d91-76412b43b05b" containerID="573894aed5622efdde63859e961b12c1441ec8c9e255c0d3ea2b009a541bc3fa" exitCode=0 Oct 31 04:27:53.444228 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:53.443804 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" event={"ID":"abc4870f-03e0-4c54-9d91-76412b43b05b","Type":"ContainerDied","Data":"573894aed5622efdde63859e961b12c1441ec8c9e255c0d3ea2b009a541bc3fa"} Oct 31 04:27:54.598479 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:54.598453 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:54.729870 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:54.729783 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc4870f-03e0-4c54-9d91-76412b43b05b-kube-api-access-c72j6\" (UniqueName: \"kubernetes.io/projected/abc4870f-03e0-4c54-9d91-76412b43b05b-kube-api-access-c72j6\") pod \"abc4870f-03e0-4c54-9d91-76412b43b05b\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " Oct 31 04:27:54.730026 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:54.729910 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-xks-ca\") pod \"abc4870f-03e0-4c54-9d91-76412b43b05b\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " Oct 31 04:27:54.730026 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:54.729946 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-openshift-ca\") pod \"abc4870f-03e0-4c54-9d91-76412b43b05b\" (UID: \"abc4870f-03e0-4c54-9d91-76412b43b05b\") " Oct 31 04:27:54.730402 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:54.730370 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-openshift-ca" pod "abc4870f-03e0-4c54-9d91-76412b43b05b" (UID: "abc4870f-03e0-4c54-9d91-76412b43b05b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:54.732103 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:54.732067 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-xks-ca" pod "abc4870f-03e0-4c54-9d91-76412b43b05b" (UID: "abc4870f-03e0-4c54-9d91-76412b43b05b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:27:54.732267 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:54.732246 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc4870f-03e0-4c54-9d91-76412b43b05b-kube-api-access-c72j6" pod "abc4870f-03e0-4c54-9d91-76412b43b05b" (UID: "abc4870f-03e0-4c54-9d91-76412b43b05b"). InnerVolumeSpecName "kube-api-access-c72j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:54.831775 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:54.831740 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:27:54.831775 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:54.831767 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c72j6\" (UniqueName: \"kubernetes.io/projected/abc4870f-03e0-4c54-9d91-76412b43b05b-kube-api-access-c72j6\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:27:54.831775 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:54.831776 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc4870f-03e0-4c54-9d91-76412b43b05b-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:27:55.457531 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:55.457496 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" Oct 31 04:27:55.457698 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:55.457531 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d" event={"ID":"abc4870f-03e0-4c54-9d91-76412b43b05b","Type":"ContainerDied","Data":"adc880bcc7af7c1821234093f329dcdb025a99f58d4a733a278b959ff09fc652"} Oct 31 04:27:55.457698 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:27:55.457581 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adc880bcc7af7c1821234093f329dcdb025a99f58d4a733a278b959ff09fc652" Oct 31 04:28:02.846226 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:28:02.846186 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:28:03.072433 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.072388 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5"] Oct 31 04:28:03.073143 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.073102 4426 cpu_manager.go:414] "containerMap: removing container" podUID="abc4870f-03e0-4c54-9d91-76412b43b05b" containerName="revoke-keys" Oct 31 04:28:03.073211 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.073145 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc4870f-03e0-4c54-9d91-76412b43b05b" containerName="revoke-keys" Oct 31 04:28:03.073301 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.073288 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc4870f-03e0-4c54-9d91-76412b43b05b" containerName="revoke-keys" Oct 31 04:28:03.076911 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.076887 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:03.106031 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.105942 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:03.106031 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.106010 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:03.106274 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.106105 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjhpx\" (UniqueName: \"kubernetes.io/projected/fa39bc32-7356-47b3-99af-18b2936b8115-kube-api-access-vjhpx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:03.130641 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.130604 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5"] Oct 31 04:28:03.172881 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.172837 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:28:03.172881 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.172860 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:28:03.173204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.173012 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:28:03.173204 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.173054 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:28:03.207012 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.206969 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:03.207216 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.207109 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:03.207306 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.207278 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhpx\" (UniqueName: \"kubernetes.io/projected/fa39bc32-7356-47b3-99af-18b2936b8115-kube-api-access-vjhpx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:03.207832 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.207789 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:03.209406 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.209389 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:03.265304 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.265269 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjhpx\" (UniqueName: \"kubernetes.io/projected/fa39bc32-7356-47b3-99af-18b2936b8115-kube-api-access-vjhpx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:03.390020 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.389983 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:03.760929 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:03.760891 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5"] Oct 31 04:28:03.763335 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:28:03.763305 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa39bc32_7356_47b3_99af_18b2936b8115.slice/crio-6396093004c292b3fd4681fb5f628972213ea13a4a93b2171aa8b5aee85bfc4f WatchSource:0}: Error finding container 6396093004c292b3fd4681fb5f628972213ea13a4a93b2171aa8b5aee85bfc4f: Status 404 returned error can't find the container with id 6396093004c292b3fd4681fb5f628972213ea13a4a93b2171aa8b5aee85bfc4f Oct 31 04:28:04.342268 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:04.342187 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:28:04.516060 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:04.515679 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" event={"ID":"fa39bc32-7356-47b3-99af-18b2936b8115","Type":"ContainerStarted","Data":"39157efd4620c20e78ee22bf61035728a3724b3c29be4ed53b7ec53c49e98a85"} Oct 31 04:28:04.516060 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:04.515746 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" event={"ID":"fa39bc32-7356-47b3-99af-18b2936b8115","Type":"ContainerStarted","Data":"6396093004c292b3fd4681fb5f628972213ea13a4a93b2171aa8b5aee85bfc4f"} Oct 31 04:28:04.626091 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:04.625970 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" podStartSLOduration=1.625955792 podStartE2EDuration="1.625955792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:04.623793862 +0000 UTC m=+3581.624419090" watchObservedRunningTime="2026-10-31 04:28:04.625955792 +0000 UTC m=+3581.626581020" Oct 31 04:28:05.523198 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:05.523161 4426 generic.go:356] "Generic (PLEG): container finished" podID="fa39bc32-7356-47b3-99af-18b2936b8115" containerID="39157efd4620c20e78ee22bf61035728a3724b3c29be4ed53b7ec53c49e98a85" exitCode=0 Oct 31 04:28:05.523600 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:05.523247 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" event={"ID":"fa39bc32-7356-47b3-99af-18b2936b8115","Type":"ContainerDied","Data":"39157efd4620c20e78ee22bf61035728a3724b3c29be4ed53b7ec53c49e98a85"} Oct 31 04:28:06.682957 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:06.682930 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:06.757811 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:06.757775 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-openshift-ca\") pod \"fa39bc32-7356-47b3-99af-18b2936b8115\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " Oct 31 04:28:06.757998 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:06.757849 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa39bc32-7356-47b3-99af-18b2936b8115-kube-api-access-vjhpx\" (UniqueName: \"kubernetes.io/projected/fa39bc32-7356-47b3-99af-18b2936b8115-kube-api-access-vjhpx\") pod \"fa39bc32-7356-47b3-99af-18b2936b8115\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " Oct 31 04:28:06.757998 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:06.757956 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-xks-ca\") pod \"fa39bc32-7356-47b3-99af-18b2936b8115\" (UID: \"fa39bc32-7356-47b3-99af-18b2936b8115\") " Oct 31 04:28:06.758203 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:06.758178 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-openshift-ca" pod "fa39bc32-7356-47b3-99af-18b2936b8115" (UID: "fa39bc32-7356-47b3-99af-18b2936b8115"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:28:06.760004 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:06.759971 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-xks-ca" pod "fa39bc32-7356-47b3-99af-18b2936b8115" (UID: "fa39bc32-7356-47b3-99af-18b2936b8115"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:28:06.760232 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:06.760040 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa39bc32-7356-47b3-99af-18b2936b8115-kube-api-access-vjhpx" pod "fa39bc32-7356-47b3-99af-18b2936b8115" (UID: "fa39bc32-7356-47b3-99af-18b2936b8115"). InnerVolumeSpecName "kube-api-access-vjhpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:28:06.859640 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:06.859554 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:28:06.859640 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:06.859586 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjhpx\" (UniqueName: \"kubernetes.io/projected/fa39bc32-7356-47b3-99af-18b2936b8115-kube-api-access-vjhpx\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:28:06.859640 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:06.859598 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa39bc32-7356-47b3-99af-18b2936b8115-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:28:07.536950 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:07.536918 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" Oct 31 04:28:07.537202 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:07.536945 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5" event={"ID":"fa39bc32-7356-47b3-99af-18b2936b8115","Type":"ContainerDied","Data":"6396093004c292b3fd4681fb5f628972213ea13a4a93b2171aa8b5aee85bfc4f"} Oct 31 04:28:07.537202 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:07.536998 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6396093004c292b3fd4681fb5f628972213ea13a4a93b2171aa8b5aee85bfc4f" Oct 31 04:28:14.467834 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:14.467792 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms"] Oct 31 04:28:14.567185 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:14.567143 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq"] Oct 31 04:28:19.469312 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:19.469266 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" podUID="9d1dacd5-6785-4091-a468-5cfcc1de41e2" containerName="payload-processing" containerID="cri-o://9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810" gracePeriod=25 Oct 31 04:28:19.568208 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:19.568121 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" podUID="08afaa1d-d616-44d0-a483-8345c229bed4" containerName="payload-pre-processing" containerID="cri-o://a4d29b981012b706aab681c55f28ae9f05477bfca17e121fabf4648232aa3b61" gracePeriod=25 Oct 31 04:28:23.359300 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:23.359262 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5"] Oct 31 04:28:23.405524 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:23.405487 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b097c5-fzll5"] Oct 31 04:28:23.838088 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:23.838051 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:28:23.842782 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:23.842759 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:28:23.894624 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:23.894585 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" podUID="9d1dacd5-6785-4091-a468-5cfcc1de41e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:28:23.894810 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:23.894594 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" podUID="08afaa1d-d616-44d0-a483-8345c229bed4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:28:25.130301 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:25.130263 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa39bc32-7356-47b3-99af-18b2936b8115" path="/var/lib/kubelet/pods/fa39bc32-7356-47b3-99af-18b2936b8115/volumes" Oct 31 04:28:25.844859 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:25.844817 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:28:32.846523 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:28:32.846485 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:28:33.894570 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:33.894529 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" podUID="08afaa1d-d616-44d0-a483-8345c229bed4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:28:33.894967 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:33.894529 4426 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" podUID="9d1dacd5-6785-4091-a468-5cfcc1de41e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:28:39.732839 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.732769 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:28:39.744713 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.744675 4426 generic.go:356] "Generic (PLEG): container finished" podID="9d1dacd5-6785-4091-a468-5cfcc1de41e2" containerID="9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810" exitCode=0 Oct 31 04:28:39.744873 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.744743 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" Oct 31 04:28:39.744873 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.744756 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" event={"ID":"9d1dacd5-6785-4091-a468-5cfcc1de41e2","Type":"ContainerDied","Data":"9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810"} Oct 31 04:28:39.744873 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.744812 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms" event={"ID":"9d1dacd5-6785-4091-a468-5cfcc1de41e2","Type":"ContainerDied","Data":"5842f35e64ab4fde90fe9a1c8f25452e0b359978838958ee3469526753887169"} Oct 31 04:28:39.744873 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.744837 4426 scope.go:122] "RemoveContainer" containerID="9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810" Oct 31 04:28:39.747792 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.747741 4426 generic.go:356] "Generic (PLEG): container finished" podID="08afaa1d-d616-44d0-a483-8345c229bed4" containerID="a4d29b981012b706aab681c55f28ae9f05477bfca17e121fabf4648232aa3b61" exitCode=0 Oct 31 04:28:39.748005 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.747826 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" event={"ID":"08afaa1d-d616-44d0-a483-8345c229bed4","Type":"ContainerDied","Data":"a4d29b981012b706aab681c55f28ae9f05477bfca17e121fabf4648232aa3b61"} Oct 31 04:28:39.756643 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.756622 4426 scope.go:122] "RemoveContainer" containerID="9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810" Oct 31 04:28:39.756940 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:28:39.756920 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810\": container with ID starting with 9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810 not found: ID does not exist" containerID="9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810" Oct 31 04:28:39.757030 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.756956 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810"} err="failed to get container status \"9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810\": rpc error: code = NotFound desc = could not find container \"9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810\": container with ID starting with 9527299dc0f816fb6079e7dd296b0cb25896530aee27ca8007d1817e24858810 not found: ID does not exist" Oct 31 04:28:39.790115 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.790067 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d1dacd5-6785-4091-a468-5cfcc1de41e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1dacd5-6785-4091-a468-5cfcc1de41e2-plugins-config-volume\") pod \"9d1dacd5-6785-4091-a468-5cfcc1de41e2\" (UID: \"9d1dacd5-6785-4091-a468-5cfcc1de41e2\") " Oct 31 04:28:39.790330 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.790199 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1dacd5-6785-4091-a468-5cfcc1de41e2-kube-api-access-6nt7v\" (UniqueName: \"kubernetes.io/projected/9d1dacd5-6785-4091-a468-5cfcc1de41e2-kube-api-access-6nt7v\") pod \"9d1dacd5-6785-4091-a468-5cfcc1de41e2\" (UID: \"9d1dacd5-6785-4091-a468-5cfcc1de41e2\") " Oct 31 04:28:39.790579 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.790551 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1dacd5-6785-4091-a468-5cfcc1de41e2-plugins-config-volume" pod "9d1dacd5-6785-4091-a468-5cfcc1de41e2" (UID: "9d1dacd5-6785-4091-a468-5cfcc1de41e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:28:39.790691 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.790671 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1dacd5-6785-4091-a468-5cfcc1de41e2-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:28:39.793554 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.793526 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1dacd5-6785-4091-a468-5cfcc1de41e2-kube-api-access-6nt7v" pod "9d1dacd5-6785-4091-a468-5cfcc1de41e2" (UID: "9d1dacd5-6785-4091-a468-5cfcc1de41e2"). InnerVolumeSpecName "kube-api-access-6nt7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:28:39.826666 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.826640 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:28:39.891297 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.891265 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08afaa1d-d616-44d0-a483-8345c229bed4-kube-api-access-6z5n8\" (UniqueName: \"kubernetes.io/projected/08afaa1d-d616-44d0-a483-8345c229bed4-kube-api-access-6z5n8\") pod \"08afaa1d-d616-44d0-a483-8345c229bed4\" (UID: \"08afaa1d-d616-44d0-a483-8345c229bed4\") " Oct 31 04:28:39.891486 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.891338 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08afaa1d-d616-44d0-a483-8345c229bed4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08afaa1d-d616-44d0-a483-8345c229bed4-plugins-config-volume\") pod \"08afaa1d-d616-44d0-a483-8345c229bed4\" (UID: \"08afaa1d-d616-44d0-a483-8345c229bed4\") " Oct 31 04:28:39.891559 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.891547 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nt7v\" (UniqueName: \"kubernetes.io/projected/9d1dacd5-6785-4091-a468-5cfcc1de41e2-kube-api-access-6nt7v\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:28:39.891748 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.891721 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08afaa1d-d616-44d0-a483-8345c229bed4-plugins-config-volume" pod "08afaa1d-d616-44d0-a483-8345c229bed4" (UID: "08afaa1d-d616-44d0-a483-8345c229bed4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:28:39.893424 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.893405 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08afaa1d-d616-44d0-a483-8345c229bed4-kube-api-access-6z5n8" pod "08afaa1d-d616-44d0-a483-8345c229bed4" (UID: "08afaa1d-d616-44d0-a483-8345c229bed4"). InnerVolumeSpecName "kube-api-access-6z5n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:28:39.993213 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.993109 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z5n8\" (UniqueName: \"kubernetes.io/projected/08afaa1d-d616-44d0-a483-8345c229bed4-kube-api-access-6z5n8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:28:39.993213 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:39.993159 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08afaa1d-d616-44d0-a483-8345c229bed4-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:28:40.161043 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:40.161006 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms"] Oct 31 04:28:40.165433 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:40.165401 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b097c5-76b9d4d59lk7ms"] Oct 31 04:28:40.756693 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:40.756609 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" Oct 31 04:28:40.756693 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:40.756621 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq" event={"ID":"08afaa1d-d616-44d0-a483-8345c229bed4","Type":"ContainerDied","Data":"56fa3e963be20ad60e9f567ba42437f897a0752cb76d2bab5abe30a34bee0cae"} Oct 31 04:28:40.756693 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:40.756685 4426 scope.go:122] "RemoveContainer" containerID="a4d29b981012b706aab681c55f28ae9f05477bfca17e121fabf4648232aa3b61" Oct 31 04:28:40.910958 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:40.910924 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq"] Oct 31 04:28:40.913543 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:40.913516 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b097c5-75576sddtq"] Oct 31 04:28:41.128647 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:41.128572 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08afaa1d-d616-44d0-a483-8345c229bed4" path="/var/lib/kubelet/pods/08afaa1d-d616-44d0-a483-8345c229bed4/volumes" Oct 31 04:28:41.128975 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:28:41.128962 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1dacd5-6785-4091-a468-5cfcc1de41e2" path="/var/lib/kubelet/pods/9d1dacd5-6785-4091-a468-5cfcc1de41e2/volumes" Oct 31 04:29:02.845845 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:29:02.845803 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:29:10.219500 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:29:10.219420 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:29:27.039502 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:29:27.039390 4426 scope.go:122] "RemoveContainer" containerID="aa1ee9ead03714aefe21ccadd1d7ef1f2a636c618be7dffc0789635c8d3df319" Oct 31 04:29:27.048610 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:29:27.048585 4426 scope.go:122] "RemoveContainer" containerID="cd70ac86812dce812ad63e7128c2b4064dc4540a6989bf4c4fabf8d933702d5d" Oct 31 04:29:32.845204 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:29:32.845166 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:29:34.602895 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:29:34.602847 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:30:02.844939 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:30:02.844892 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:30:09.131565 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:09.131526 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:30:18.263397 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.263364 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64"] Oct 31 04:30:18.263810 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.263797 4426 cpu_manager.go:414] "containerMap: removing container" podUID="9d1dacd5-6785-4091-a468-5cfcc1de41e2" containerName="payload-processing" Oct 31 04:30:18.263856 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.263810 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1dacd5-6785-4091-a468-5cfcc1de41e2" containerName="payload-processing" Oct 31 04:30:18.263856 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.263832 4426 cpu_manager.go:414] "containerMap: removing container" podUID="fa39bc32-7356-47b3-99af-18b2936b8115" containerName="revoke-keys" Oct 31 04:30:18.263856 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.263838 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa39bc32-7356-47b3-99af-18b2936b8115" containerName="revoke-keys" Oct 31 04:30:18.263856 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.263850 4426 cpu_manager.go:414] "containerMap: removing container" podUID="08afaa1d-d616-44d0-a483-8345c229bed4" containerName="payload-pre-processing" Oct 31 04:30:18.263856 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.263856 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08afaa1d-d616-44d0-a483-8345c229bed4" containerName="payload-pre-processing" Oct 31 04:30:18.264030 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.263917 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa39bc32-7356-47b3-99af-18b2936b8115" containerName="revoke-keys" Oct 31 04:30:18.264030 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.263928 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="08afaa1d-d616-44d0-a483-8345c229bed4" containerName="payload-pre-processing" Oct 31 04:30:18.264030 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.263936 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1dacd5-6785-4091-a468-5cfcc1de41e2" containerName="payload-processing" Oct 31 04:30:18.267351 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.267333 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:18.362660 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.362613 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bft64\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:18.362876 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.362784 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92slx\" (UniqueName: \"kubernetes.io/projected/a2701903-0553-4b83-b607-291c2c5e15b1-kube-api-access-92slx\") pod \"maas-api-revoke-sub-452b6d7b-bft64\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:18.362876 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.362837 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bft64\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:18.365740 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.365713 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:30:18.365822 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.365736 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78tsm\"" Oct 31 04:30:18.365822 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.365720 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:30:18.365946 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.365822 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:30:18.368878 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.368853 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64"] Oct 31 04:30:18.464273 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.464235 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bft64\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:18.464458 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.464361 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92slx\" (UniqueName: \"kubernetes.io/projected/a2701903-0553-4b83-b607-291c2c5e15b1-kube-api-access-92slx\") pod \"maas-api-revoke-sub-452b6d7b-bft64\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:18.464458 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.464400 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bft64\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:18.465055 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.465027 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bft64\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:18.466740 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.466715 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bft64\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:18.522664 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.522582 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92slx\" (UniqueName: \"kubernetes.io/projected/a2701903-0553-4b83-b607-291c2c5e15b1-kube-api-access-92slx\") pod \"maas-api-revoke-sub-452b6d7b-bft64\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:18.580787 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.580748 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:18.953344 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:18.953309 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64"] Oct 31 04:30:18.960231 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:30:18.960191 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2701903_0553_4b83_b607_291c2c5e15b1.slice/crio-8d9ea1f9fd8d518d7d1785b5314189149d29ded88c43e68326085ea51ed52152 WatchSource:0}: Error finding container 8d9ea1f9fd8d518d7d1785b5314189149d29ded88c43e68326085ea51ed52152: Status 404 returned error can't find the container with id 8d9ea1f9fd8d518d7d1785b5314189149d29ded88c43e68326085ea51ed52152 Oct 31 04:30:19.440432 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:19.440396 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" event={"ID":"a2701903-0553-4b83-b607-291c2c5e15b1","Type":"ContainerStarted","Data":"3eea4a39bd399417b1570ab28cce133b7040debdb1f741c95c4cac3e9c75dd49"} Oct 31 04:30:19.440834 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:19.440452 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" event={"ID":"a2701903-0553-4b83-b607-291c2c5e15b1","Type":"ContainerStarted","Data":"8d9ea1f9fd8d518d7d1785b5314189149d29ded88c43e68326085ea51ed52152"} Oct 31 04:30:19.550383 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:19.550277 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" podStartSLOduration=1.550261487 podStartE2EDuration="1.550261487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:19.548776152 +0000 UTC m=+3716.549401380" watchObservedRunningTime="2026-10-31 04:30:19.550261487 +0000 UTC m=+3716.550886718" Oct 31 04:30:20.448118 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:20.448081 4426 generic.go:356] "Generic (PLEG): container finished" podID="a2701903-0553-4b83-b607-291c2c5e15b1" containerID="3eea4a39bd399417b1570ab28cce133b7040debdb1f741c95c4cac3e9c75dd49" exitCode=0 Oct 31 04:30:20.448540 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:20.448144 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" event={"ID":"a2701903-0553-4b83-b607-291c2c5e15b1","Type":"ContainerDied","Data":"3eea4a39bd399417b1570ab28cce133b7040debdb1f741c95c4cac3e9c75dd49"} Oct 31 04:30:21.596141 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:21.596101 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:21.698106 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:21.698069 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-openshift-ca\") pod \"a2701903-0553-4b83-b607-291c2c5e15b1\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " Oct 31 04:30:21.698312 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:21.698156 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2701903-0553-4b83-b607-291c2c5e15b1-kube-api-access-92slx\" (UniqueName: \"kubernetes.io/projected/a2701903-0553-4b83-b607-291c2c5e15b1-kube-api-access-92slx\") pod \"a2701903-0553-4b83-b607-291c2c5e15b1\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " Oct 31 04:30:21.698312 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:21.698257 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-xks-ca\") pod \"a2701903-0553-4b83-b607-291c2c5e15b1\" (UID: \"a2701903-0553-4b83-b607-291c2c5e15b1\") " Oct 31 04:30:21.698488 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:21.698465 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-openshift-ca" pod "a2701903-0553-4b83-b607-291c2c5e15b1" (UID: "a2701903-0553-4b83-b607-291c2c5e15b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:21.698623 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:21.698603 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:30:21.700397 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:21.700371 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-xks-ca" pod "a2701903-0553-4b83-b607-291c2c5e15b1" (UID: "a2701903-0553-4b83-b607-291c2c5e15b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:21.700490 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:21.700399 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2701903-0553-4b83-b607-291c2c5e15b1-kube-api-access-92slx" pod "a2701903-0553-4b83-b607-291c2c5e15b1" (UID: "a2701903-0553-4b83-b607-291c2c5e15b1"). InnerVolumeSpecName "kube-api-access-92slx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:21.800046 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:21.800008 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92slx\" (UniqueName: \"kubernetes.io/projected/a2701903-0553-4b83-b607-291c2c5e15b1-kube-api-access-92slx\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:30:21.800046 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:21.800041 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2701903-0553-4b83-b607-291c2c5e15b1-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:30:22.461677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:22.461640 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" event={"ID":"a2701903-0553-4b83-b607-291c2c5e15b1","Type":"ContainerDied","Data":"8d9ea1f9fd8d518d7d1785b5314189149d29ded88c43e68326085ea51ed52152"} Oct 31 04:30:22.461677 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:22.461661 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64" Oct 31 04:30:22.461897 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:22.461689 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d9ea1f9fd8d518d7d1785b5314189149d29ded88c43e68326085ea51ed52152" Oct 31 04:30:32.846257 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:30:32.846212 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:30:34.638438 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:34.638397 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:30:49.230471 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:49.230434 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:30:57.402207 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:30:57.402175 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:31:02.848824 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:31:02.848779 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:31:06.905575 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:06.905529 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:31:26.527452 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:26.527414 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:31:32.848496 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:31:32.848447 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:31:41.782366 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.782327 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h"] Oct 31 04:31:41.782886 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.782800 4426 cpu_manager.go:414] "containerMap: removing container" podUID="a2701903-0553-4b83-b607-291c2c5e15b1" containerName="revoke-keys" Oct 31 04:31:41.782886 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.782814 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2701903-0553-4b83-b607-291c2c5e15b1" containerName="revoke-keys" Oct 31 04:31:41.782984 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.782914 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2701903-0553-4b83-b607-291c2c5e15b1" containerName="revoke-keys" Oct 31 04:31:41.786383 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.786362 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:31:41.855563 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.843850 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h"] Oct 31 04:31:41.872273 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.872235 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd7b138-fec6-492b-845f-460d90f5c0c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h\" (UID: \"1fd7b138-fec6-492b-845f-460d90f5c0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:31:41.873706 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.873669 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hw9z\" (UniqueName: \"kubernetes.io/projected/1fd7b138-fec6-492b-845f-460d90f5c0c4-kube-api-access-6hw9z\") pod \"payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h\" (UID: \"1fd7b138-fec6-492b-845f-460d90f5c0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:31:41.881849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.881769 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-40966d85\"" Oct 31 04:31:41.882942 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.881797 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-40966d85-dockercfg-88nfj\"" Oct 31 04:31:41.940509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.940472 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292"] Oct 31 04:31:41.944157 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.944102 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:31:41.974856 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.974820 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd7b138-fec6-492b-845f-460d90f5c0c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h\" (UID: \"1fd7b138-fec6-492b-845f-460d90f5c0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:31:41.975045 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.974937 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hw9z\" (UniqueName: \"kubernetes.io/projected/1fd7b138-fec6-492b-845f-460d90f5c0c4-kube-api-access-6hw9z\") pod \"payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h\" (UID: \"1fd7b138-fec6-492b-845f-460d90f5c0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:31:41.975625 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:41.975599 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd7b138-fec6-492b-845f-460d90f5c0c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h\" (UID: \"1fd7b138-fec6-492b-845f-460d90f5c0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:31:42.001396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.001357 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292"] Oct 31 04:31:42.032891 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.032812 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hw9z\" (UniqueName: \"kubernetes.io/projected/1fd7b138-fec6-492b-845f-460d90f5c0c4-kube-api-access-6hw9z\") pod \"payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h\" (UID: \"1fd7b138-fec6-492b-845f-460d90f5c0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:31:42.076444 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.076409 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee92a62-7b2e-4ef2-86db-3cac60c51139-plugins-config-volume\") pod \"payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292\" (UID: \"aee92a62-7b2e-4ef2-86db-3cac60c51139\") " pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:31:42.076618 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.076453 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6zm6\" (UniqueName: \"kubernetes.io/projected/aee92a62-7b2e-4ef2-86db-3cac60c51139-kube-api-access-n6zm6\") pod \"payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292\" (UID: \"aee92a62-7b2e-4ef2-86db-3cac60c51139\") " pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:31:42.101897 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.101862 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:31:42.177748 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.177710 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee92a62-7b2e-4ef2-86db-3cac60c51139-plugins-config-volume\") pod \"payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292\" (UID: \"aee92a62-7b2e-4ef2-86db-3cac60c51139\") " pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:31:42.177898 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.177773 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6zm6\" (UniqueName: \"kubernetes.io/projected/aee92a62-7b2e-4ef2-86db-3cac60c51139-kube-api-access-n6zm6\") pod \"payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292\" (UID: \"aee92a62-7b2e-4ef2-86db-3cac60c51139\") " pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:31:42.178856 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.178829 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee92a62-7b2e-4ef2-86db-3cac60c51139-plugins-config-volume\") pod \"payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292\" (UID: \"aee92a62-7b2e-4ef2-86db-3cac60c51139\") " pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:31:42.235878 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.235821 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6zm6\" (UniqueName: \"kubernetes.io/projected/aee92a62-7b2e-4ef2-86db-3cac60c51139-kube-api-access-n6zm6\") pod \"payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292\" (UID: \"aee92a62-7b2e-4ef2-86db-3cac60c51139\") " pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:31:42.257700 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.257664 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:31:42.474702 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.474667 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h"] Oct 31 04:31:42.481071 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:31:42.481042 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd7b138_fec6_492b_845f_460d90f5c0c4.slice/crio-98740f1558f67874f61cd56f509196f57de5987ee6a38392b28d0b615c00d0c6 WatchSource:0}: Error finding container 98740f1558f67874f61cd56f509196f57de5987ee6a38392b28d0b615c00d0c6: Status 404 returned error can't find the container with id 98740f1558f67874f61cd56f509196f57de5987ee6a38392b28d0b615c00d0c6 Oct 31 04:31:42.632822 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:42.632792 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292"] Oct 31 04:31:42.639548 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:31:42.639517 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee92a62_7b2e_4ef2_86db_3cac60c51139.slice/crio-457896a32b23e8ef68285e8d6fcade34eece46e755769caa8b0c56ffa077bf3f WatchSource:0}: Error finding container 457896a32b23e8ef68285e8d6fcade34eece46e755769caa8b0c56ffa077bf3f: Status 404 returned error can't find the container with id 457896a32b23e8ef68285e8d6fcade34eece46e755769caa8b0c56ffa077bf3f Oct 31 04:31:43.068309 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:43.068270 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" event={"ID":"aee92a62-7b2e-4ef2-86db-3cac60c51139","Type":"ContainerStarted","Data":"894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc"} Oct 31 04:31:43.068767 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:43.068336 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" event={"ID":"aee92a62-7b2e-4ef2-86db-3cac60c51139","Type":"ContainerStarted","Data":"457896a32b23e8ef68285e8d6fcade34eece46e755769caa8b0c56ffa077bf3f"} Oct 31 04:31:43.068767 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:43.068488 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:31:43.070060 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:43.070031 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" event={"ID":"1fd7b138-fec6-492b-845f-460d90f5c0c4","Type":"ContainerStarted","Data":"5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24"} Oct 31 04:31:43.070218 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:43.070078 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" event={"ID":"1fd7b138-fec6-492b-845f-460d90f5c0c4","Type":"ContainerStarted","Data":"98740f1558f67874f61cd56f509196f57de5987ee6a38392b28d0b615c00d0c6"} Oct 31 04:31:43.070218 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:43.070109 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:31:43.176434 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:43.176375 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" podStartSLOduration=2.17636238 podStartE2EDuration="2.17636238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:43.176088771 +0000 UTC m=+3800.176714000" watchObservedRunningTime="2026-10-31 04:31:43.17636238 +0000 UTC m=+3800.176987607" Oct 31 04:31:43.291641 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:43.291582 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" podStartSLOduration=2.2915666740000002 podStartE2EDuration="2.291566674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:43.288524905 +0000 UTC m=+3800.289150134" watchObservedRunningTime="2026-10-31 04:31:43.291566674 +0000 UTC m=+3800.292191929" Oct 31 04:31:49.133802 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:49.133767 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:31:54.079589 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:54.079557 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:31:54.080095 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:31:54.079639 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:32:02.846906 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:32:02.846865 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:32:19.413433 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.413394 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z"] Oct 31 04:32:19.417382 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.417352 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:19.515743 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.515708 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:32:19.515743 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.515727 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:32:19.515743 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.515733 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:32:19.516053 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.515709 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgjbl\"" Oct 31 04:32:19.518676 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.518649 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z"] Oct 31 04:32:19.541390 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.541361 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-40966d85-58b7z\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:19.541390 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.541391 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thfsv\" (UniqueName: \"kubernetes.io/projected/75c4d379-6cc2-487a-bf0a-0e64d0c09535-kube-api-access-thfsv\") pod \"maas-api-revoke-keys-e2e-ait-40966d85-58b7z\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:19.541566 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.541418 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-40966d85-58b7z\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:19.642242 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.642198 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-40966d85-58b7z\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:19.642242 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.642246 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thfsv\" (UniqueName: \"kubernetes.io/projected/75c4d379-6cc2-487a-bf0a-0e64d0c09535-kube-api-access-thfsv\") pod \"maas-api-revoke-keys-e2e-ait-40966d85-58b7z\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:19.642505 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.642282 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-40966d85-58b7z\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:19.642908 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.642886 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-40966d85-58b7z\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:19.644775 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.644752 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-40966d85-58b7z\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:19.700181 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.700071 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thfsv\" (UniqueName: \"kubernetes.io/projected/75c4d379-6cc2-487a-bf0a-0e64d0c09535-kube-api-access-thfsv\") pod \"maas-api-revoke-keys-e2e-ait-40966d85-58b7z\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:19.731498 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:19.731468 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:20.108509 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:20.108424 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z"] Oct 31 04:32:20.115431 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:32:20.115397 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c4d379_6cc2_487a_bf0a_0e64d0c09535.slice/crio-5822c514dca5c8d84804fb6d802777dec4e64b5e0d67e54350330bcc803bb0e1 WatchSource:0}: Error finding container 5822c514dca5c8d84804fb6d802777dec4e64b5e0d67e54350330bcc803bb0e1: Status 404 returned error can't find the container with id 5822c514dca5c8d84804fb6d802777dec4e64b5e0d67e54350330bcc803bb0e1 Oct 31 04:32:20.300963 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:20.300926 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" event={"ID":"75c4d379-6cc2-487a-bf0a-0e64d0c09535","Type":"ContainerStarted","Data":"5822c514dca5c8d84804fb6d802777dec4e64b5e0d67e54350330bcc803bb0e1"} Oct 31 04:32:21.308818 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:21.308723 4426 generic.go:356] "Generic (PLEG): container finished" podID="75c4d379-6cc2-487a-bf0a-0e64d0c09535" containerID="2fd15e75fb52f932dd703afe5c5bbc79205e5f211e3c9cec306cbad6372e15db" exitCode=0 Oct 31 04:32:21.308818 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:21.308763 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" event={"ID":"75c4d379-6cc2-487a-bf0a-0e64d0c09535","Type":"ContainerDied","Data":"2fd15e75fb52f932dd703afe5c5bbc79205e5f211e3c9cec306cbad6372e15db"} Oct 31 04:32:22.455854 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:22.455826 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:22.572304 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:22.572215 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c4d379-6cc2-487a-bf0a-0e64d0c09535-kube-api-access-thfsv\" (UniqueName: \"kubernetes.io/projected/75c4d379-6cc2-487a-bf0a-0e64d0c09535-kube-api-access-thfsv\") pod \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " Oct 31 04:32:22.572304 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:22.572280 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-openshift-ca\") pod \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " Oct 31 04:32:22.572535 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:22.572376 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-xks-ca\") pod \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\" (UID: \"75c4d379-6cc2-487a-bf0a-0e64d0c09535\") " Oct 31 04:32:22.572779 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:22.572741 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-openshift-ca" pod "75c4d379-6cc2-487a-bf0a-0e64d0c09535" (UID: "75c4d379-6cc2-487a-bf0a-0e64d0c09535"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:22.574712 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:22.574683 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c4d379-6cc2-487a-bf0a-0e64d0c09535-kube-api-access-thfsv" pod "75c4d379-6cc2-487a-bf0a-0e64d0c09535" (UID: "75c4d379-6cc2-487a-bf0a-0e64d0c09535"). InnerVolumeSpecName "kube-api-access-thfsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:22.574977 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:22.574951 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-xks-ca" pod "75c4d379-6cc2-487a-bf0a-0e64d0c09535" (UID: "75c4d379-6cc2-487a-bf0a-0e64d0c09535"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:22.673920 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:22.673880 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thfsv\" (UniqueName: \"kubernetes.io/projected/75c4d379-6cc2-487a-bf0a-0e64d0c09535-kube-api-access-thfsv\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:32:22.673920 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:22.673914 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:32:22.673920 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:22.673925 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c4d379-6cc2-487a-bf0a-0e64d0c09535-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:32:23.322926 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:23.322897 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" Oct 31 04:32:23.323078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:23.322934 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z" event={"ID":"75c4d379-6cc2-487a-bf0a-0e64d0c09535","Type":"ContainerDied","Data":"5822c514dca5c8d84804fb6d802777dec4e64b5e0d67e54350330bcc803bb0e1"} Oct 31 04:32:23.323078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:23.322985 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5822c514dca5c8d84804fb6d802777dec4e64b5e0d67e54350330bcc803bb0e1" Oct 31 04:32:30.848633 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:30.848593 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292"] Oct 31 04:32:30.949863 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:30.949828 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h"] Oct 31 04:32:32.847734 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:32:32.847697 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:32:35.849481 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:35.849431 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" podUID="aee92a62-7b2e-4ef2-86db-3cac60c51139" containerName="payload-processing" containerID="cri-o://894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc" gracePeriod=25 Oct 31 04:32:35.951135 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:35.951082 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" podUID="1fd7b138-fec6-492b-845f-460d90f5c0c4" containerName="payload-pre-processing" containerID="cri-o://5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24" gracePeriod=25 Oct 31 04:32:36.104472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.104398 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:32:36.108814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.108783 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee92a62-7b2e-4ef2-86db-3cac60c51139-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee92a62-7b2e-4ef2-86db-3cac60c51139-plugins-config-volume\") pod \"aee92a62-7b2e-4ef2-86db-3cac60c51139\" (UID: \"aee92a62-7b2e-4ef2-86db-3cac60c51139\") " Oct 31 04:32:36.109224 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.109195 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee92a62-7b2e-4ef2-86db-3cac60c51139-plugins-config-volume" pod "aee92a62-7b2e-4ef2-86db-3cac60c51139" (UID: "aee92a62-7b2e-4ef2-86db-3cac60c51139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:36.208576 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.208550 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:32:36.209715 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.209695 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd7b138-fec6-492b-845f-460d90f5c0c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd7b138-fec6-492b-845f-460d90f5c0c4-plugins-config-volume\") pod \"1fd7b138-fec6-492b-845f-460d90f5c0c4\" (UID: \"1fd7b138-fec6-492b-845f-460d90f5c0c4\") " Oct 31 04:32:36.209776 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.209731 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd7b138-fec6-492b-845f-460d90f5c0c4-kube-api-access-6hw9z\" (UniqueName: \"kubernetes.io/projected/1fd7b138-fec6-492b-845f-460d90f5c0c4-kube-api-access-6hw9z\") pod \"1fd7b138-fec6-492b-845f-460d90f5c0c4\" (UID: \"1fd7b138-fec6-492b-845f-460d90f5c0c4\") " Oct 31 04:32:36.209815 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.209796 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee92a62-7b2e-4ef2-86db-3cac60c51139-kube-api-access-n6zm6\" (UniqueName: \"kubernetes.io/projected/aee92a62-7b2e-4ef2-86db-3cac60c51139-kube-api-access-n6zm6\") pod \"aee92a62-7b2e-4ef2-86db-3cac60c51139\" (UID: \"aee92a62-7b2e-4ef2-86db-3cac60c51139\") " Oct 31 04:32:36.210150 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.210098 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd7b138-fec6-492b-845f-460d90f5c0c4-plugins-config-volume" pod "1fd7b138-fec6-492b-845f-460d90f5c0c4" (UID: "1fd7b138-fec6-492b-845f-460d90f5c0c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:36.210269 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.210189 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee92a62-7b2e-4ef2-86db-3cac60c51139-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:32:36.211870 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.211851 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd7b138-fec6-492b-845f-460d90f5c0c4-kube-api-access-6hw9z" pod "1fd7b138-fec6-492b-845f-460d90f5c0c4" (UID: "1fd7b138-fec6-492b-845f-460d90f5c0c4"). InnerVolumeSpecName "kube-api-access-6hw9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:36.211924 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.211881 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee92a62-7b2e-4ef2-86db-3cac60c51139-kube-api-access-n6zm6" pod "aee92a62-7b2e-4ef2-86db-3cac60c51139" (UID: "aee92a62-7b2e-4ef2-86db-3cac60c51139"). InnerVolumeSpecName "kube-api-access-n6zm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:36.311527 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.311485 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd7b138-fec6-492b-845f-460d90f5c0c4-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:32:36.311527 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.311516 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hw9z\" (UniqueName: \"kubernetes.io/projected/1fd7b138-fec6-492b-845f-460d90f5c0c4-kube-api-access-6hw9z\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:32:36.311527 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.311526 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6zm6\" (UniqueName: \"kubernetes.io/projected/aee92a62-7b2e-4ef2-86db-3cac60c51139-kube-api-access-n6zm6\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:32:36.416244 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.416204 4426 generic.go:356] "Generic (PLEG): container finished" podID="1fd7b138-fec6-492b-845f-460d90f5c0c4" containerID="5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24" exitCode=0 Oct 31 04:32:36.416457 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.416274 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" Oct 31 04:32:36.416457 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.416372 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" event={"ID":"1fd7b138-fec6-492b-845f-460d90f5c0c4","Type":"ContainerDied","Data":"5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24"} Oct 31 04:32:36.416457 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.416432 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h" event={"ID":"1fd7b138-fec6-492b-845f-460d90f5c0c4","Type":"ContainerDied","Data":"98740f1558f67874f61cd56f509196f57de5987ee6a38392b28d0b615c00d0c6"} Oct 31 04:32:36.416652 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.416458 4426 scope.go:122] "RemoveContainer" containerID="5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24" Oct 31 04:32:36.418751 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.418720 4426 generic.go:356] "Generic (PLEG): container finished" podID="aee92a62-7b2e-4ef2-86db-3cac60c51139" containerID="894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc" exitCode=0 Oct 31 04:32:36.418921 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.418811 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" event={"ID":"aee92a62-7b2e-4ef2-86db-3cac60c51139","Type":"ContainerDied","Data":"894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc"} Oct 31 04:32:36.418921 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.418857 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" event={"ID":"aee92a62-7b2e-4ef2-86db-3cac60c51139","Type":"ContainerDied","Data":"457896a32b23e8ef68285e8d6fcade34eece46e755769caa8b0c56ffa077bf3f"} Oct 31 04:32:36.419053 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.418926 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292" Oct 31 04:32:36.428671 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.428653 4426 scope.go:122] "RemoveContainer" containerID="5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24" Oct 31 04:32:36.428948 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:32:36.428921 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24\": container with ID starting with 5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24 not found: ID does not exist" containerID="5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24" Oct 31 04:32:36.429077 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.428954 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24"} err="failed to get container status \"5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24\": rpc error: code = NotFound desc = could not find container \"5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24\": container with ID starting with 5d4419032d54735abade3814ee6e5acc5bfb080e62609c2f6892bb6965832b24 not found: ID does not exist" Oct 31 04:32:36.429077 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.428969 4426 scope.go:122] "RemoveContainer" containerID="894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc" Oct 31 04:32:36.437624 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.437604 4426 scope.go:122] "RemoveContainer" containerID="894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc" Oct 31 04:32:36.437914 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:32:36.437896 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc\": container with ID starting with 894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc not found: ID does not exist" containerID="894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc" Oct 31 04:32:36.437956 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.437922 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc"} err="failed to get container status \"894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc\": rpc error: code = NotFound desc = could not find container \"894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc\": container with ID starting with 894676e37483813cc226f6e100a474033507d3e871c80f7bd13d844996d1b3cc not found: ID does not exist" Oct 31 04:32:36.573586 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.573548 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292"] Oct 31 04:32:36.577250 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.577220 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40966d85-7df7d8ddc9-f6292"] Oct 31 04:32:36.678553 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.678474 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h"] Oct 31 04:32:36.684925 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:36.684892 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40966d85-67b74c58f5-hxj5h"] Oct 31 04:32:37.128825 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:37.128743 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd7b138-fec6-492b-845f-460d90f5c0c4" path="/var/lib/kubelet/pods/1fd7b138-fec6-492b-845f-460d90f5c0c4/volumes" Oct 31 04:32:37.129271 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:37.129115 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee92a62-7b2e-4ef2-86db-3cac60c51139" path="/var/lib/kubelet/pods/aee92a62-7b2e-4ef2-86db-3cac60c51139/volumes" Oct 31 04:32:38.339405 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:38.339370 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z"] Oct 31 04:32:38.383090 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:38.383059 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40966d85-58b7z"] Oct 31 04:32:39.129216 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:39.129179 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c4d379-6cc2-487a-bf0a-0e64d0c09535" path="/var/lib/kubelet/pods/75c4d379-6cc2-487a-bf0a-0e64d0c09535/volumes" Oct 31 04:32:40.698340 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:40.698305 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:32:55.046477 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:55.046426 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d"] Oct 31 04:32:55.092913 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:55.092874 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acd39fd9-q279d"] Oct 31 04:32:55.128433 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:55.128398 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc4870f-03e0-4c54-9d91-76412b43b05b" path="/var/lib/kubelet/pods/abc4870f-03e0-4c54-9d91-76412b43b05b/volumes" Oct 31 04:32:59.745552 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.745513 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f"] Oct 31 04:32:59.745980 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.745964 4426 cpu_manager.go:414] "containerMap: removing container" podUID="1fd7b138-fec6-492b-845f-460d90f5c0c4" containerName="payload-pre-processing" Oct 31 04:32:59.746029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.745981 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd7b138-fec6-492b-845f-460d90f5c0c4" containerName="payload-pre-processing" Oct 31 04:32:59.746029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.746001 4426 cpu_manager.go:414] "containerMap: removing container" podUID="aee92a62-7b2e-4ef2-86db-3cac60c51139" containerName="payload-processing" Oct 31 04:32:59.746029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.746008 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee92a62-7b2e-4ef2-86db-3cac60c51139" containerName="payload-processing" Oct 31 04:32:59.746029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.746020 4426 cpu_manager.go:414] "containerMap: removing container" podUID="75c4d379-6cc2-487a-bf0a-0e64d0c09535" containerName="revoke-keys" Oct 31 04:32:59.746029 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.746025 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c4d379-6cc2-487a-bf0a-0e64d0c09535" containerName="revoke-keys" Oct 31 04:32:59.746225 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.746096 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd7b138-fec6-492b-845f-460d90f5c0c4" containerName="payload-pre-processing" Oct 31 04:32:59.746225 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.746106 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee92a62-7b2e-4ef2-86db-3cac60c51139" containerName="payload-processing" Oct 31 04:32:59.746225 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.746112 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c4d379-6cc2-487a-bf0a-0e64d0c09535" containerName="revoke-keys" Oct 31 04:32:59.751012 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.750984 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:32:59.848611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.848574 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-bb6bc8-dockercfg-9ppjs\"" Oct 31 04:32:59.848611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.848574 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-bb6bc8\"" Oct 31 04:32:59.850477 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.850429 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773db981-cc62-4f1d-af35-7d09a4551e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f\" (UID: \"773db981-cc62-4f1d-af35-7d09a4551e23\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:32:59.851009 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.850983 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vsb8\" (UniqueName: \"kubernetes.io/projected/773db981-cc62-4f1d-af35-7d09a4551e23-kube-api-access-7vsb8\") pod \"payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f\" (UID: \"773db981-cc62-4f1d-af35-7d09a4551e23\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:32:59.871849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.864566 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f"] Oct 31 04:32:59.911084 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.911038 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs"] Oct 31 04:32:59.914892 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.914872 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:32:59.952493 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.952436 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773db981-cc62-4f1d-af35-7d09a4551e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f\" (UID: \"773db981-cc62-4f1d-af35-7d09a4551e23\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:32:59.952697 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.952516 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vsb8\" (UniqueName: \"kubernetes.io/projected/773db981-cc62-4f1d-af35-7d09a4551e23-kube-api-access-7vsb8\") pod \"payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f\" (UID: \"773db981-cc62-4f1d-af35-7d09a4551e23\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:32:59.953267 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.953244 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773db981-cc62-4f1d-af35-7d09a4551e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f\" (UID: \"773db981-cc62-4f1d-af35-7d09a4551e23\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:32:59.969857 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:32:59.969818 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs"] Oct 31 04:33:00.010318 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.010238 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vsb8\" (UniqueName: \"kubernetes.io/projected/773db981-cc62-4f1d-af35-7d09a4551e23-kube-api-access-7vsb8\") pod \"payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f\" (UID: \"773db981-cc62-4f1d-af35-7d09a4551e23\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:33:00.053195 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.053156 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41758200-8479-4fbb-9dd4-379fb66153fc-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs\" (UID: \"41758200-8479-4fbb-9dd4-379fb66153fc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:33:00.053404 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.053204 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-799x9\" (UniqueName: \"kubernetes.io/projected/41758200-8479-4fbb-9dd4-379fb66153fc-kube-api-access-799x9\") pod \"payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs\" (UID: \"41758200-8479-4fbb-9dd4-379fb66153fc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:33:00.064494 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.064463 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:33:00.154050 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.154008 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41758200-8479-4fbb-9dd4-379fb66153fc-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs\" (UID: \"41758200-8479-4fbb-9dd4-379fb66153fc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:33:00.154050 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.154055 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-799x9\" (UniqueName: \"kubernetes.io/projected/41758200-8479-4fbb-9dd4-379fb66153fc-kube-api-access-799x9\") pod \"payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs\" (UID: \"41758200-8479-4fbb-9dd4-379fb66153fc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:33:00.154881 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.154858 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41758200-8479-4fbb-9dd4-379fb66153fc-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs\" (UID: \"41758200-8479-4fbb-9dd4-379fb66153fc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:33:00.212625 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.212579 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-799x9\" (UniqueName: \"kubernetes.io/projected/41758200-8479-4fbb-9dd4-379fb66153fc-kube-api-access-799x9\") pod \"payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs\" (UID: \"41758200-8479-4fbb-9dd4-379fb66153fc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:33:00.228764 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.228729 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:33:00.440538 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.440497 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f"] Oct 31 04:33:00.446994 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:33:00.446969 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773db981_cc62_4f1d_af35_7d09a4551e23.slice/crio-7f38b4934340918446499c20339a105ff0069d06b3b08874acff713eac609f82 WatchSource:0}: Error finding container 7f38b4934340918446499c20339a105ff0069d06b3b08874acff713eac609f82: Status 404 returned error can't find the container with id 7f38b4934340918446499c20339a105ff0069d06b3b08874acff713eac609f82 Oct 31 04:33:00.594034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.593943 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" event={"ID":"773db981-cc62-4f1d-af35-7d09a4551e23","Type":"ContainerStarted","Data":"cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848"} Oct 31 04:33:00.594034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.593997 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" event={"ID":"773db981-cc62-4f1d-af35-7d09a4551e23","Type":"ContainerStarted","Data":"7f38b4934340918446499c20339a105ff0069d06b3b08874acff713eac609f82"} Oct 31 04:33:00.594034 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.594023 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:33:00.603461 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.603435 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs"] Oct 31 04:33:00.609692 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:33:00.609665 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41758200_8479_4fbb_9dd4_379fb66153fc.slice/crio-5d62a4fac841e18a86d9ebcaf9d728097b3ae198db55a1637c6ee19bc6e4b944 WatchSource:0}: Error finding container 5d62a4fac841e18a86d9ebcaf9d728097b3ae198db55a1637c6ee19bc6e4b944: Status 404 returned error can't find the container with id 5d62a4fac841e18a86d9ebcaf9d728097b3ae198db55a1637c6ee19bc6e4b944 Oct 31 04:33:00.705684 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:00.705625 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" podStartSLOduration=1.7056103440000001 podStartE2EDuration="1.705610344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:00.703513613 +0000 UTC m=+3877.704138842" watchObservedRunningTime="2026-10-31 04:33:00.705610344 +0000 UTC m=+3877.706235621" Oct 31 04:33:01.603784 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:01.603254 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" event={"ID":"41758200-8479-4fbb-9dd4-379fb66153fc","Type":"ContainerStarted","Data":"1b2009745fa12df88f8ea10126e932797973d869dfe20fe1fbfc5850270ba37a"} Oct 31 04:33:01.603784 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:01.603360 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" event={"ID":"41758200-8479-4fbb-9dd4-379fb66153fc","Type":"ContainerStarted","Data":"5d62a4fac841e18a86d9ebcaf9d728097b3ae198db55a1637c6ee19bc6e4b944"} Oct 31 04:33:01.605032 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:01.605007 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:33:01.716282 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:01.716218 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" podStartSLOduration=2.71619277 podStartE2EDuration="2.71619277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:01.713654249 +0000 UTC m=+3878.714279474" watchObservedRunningTime="2026-10-31 04:33:01.71619277 +0000 UTC m=+3878.716817998" Oct 31 04:33:02.851211 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:33:02.851171 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:33:09.843261 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:09.843222 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:33:11.606405 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:11.606377 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:33:13.618707 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:13.618622 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:33:23.884943 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:23.884911 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:33:23.891788 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:33:23.891762 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:33:32.845973 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:33:32.845934 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:34:02.845464 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:34:02.845422 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:34:27.137082 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:27.136974 4426 scope.go:122] "RemoveContainer" containerID="39157efd4620c20e78ee22bf61035728a3724b3c29be4ed53b7ec53c49e98a85" Oct 31 04:34:27.146372 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:27.146350 4426 scope.go:122] "RemoveContainer" containerID="573894aed5622efdde63859e961b12c1441ec8c9e255c0d3ea2b009a541bc3fa" Oct 31 04:34:32.845945 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:34:32.845901 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:34:51.817462 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.817411 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz"] Oct 31 04:34:51.821373 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.821347 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:51.852460 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.852424 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkrtz\" (UniqueName: \"kubernetes.io/projected/2006ea02-e4d8-4d85-9b3d-973043279153-kube-api-access-gkrtz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:51.852634 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.852568 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:51.852684 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.852640 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:51.872741 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.872707 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:34:51.872937 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.872710 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:34:51.877090 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.877057 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz"] Oct 31 04:34:51.896614 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.896585 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:34:51.918889 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.918862 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgjbl\"" Oct 31 04:34:51.953833 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.953796 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:51.954045 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.953851 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkrtz\" (UniqueName: \"kubernetes.io/projected/2006ea02-e4d8-4d85-9b3d-973043279153-kube-api-access-gkrtz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:51.954045 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.953932 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:51.954559 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.954532 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:51.956433 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:51.956410 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:52.011307 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:52.011280 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkrtz\" (UniqueName: \"kubernetes.io/projected/2006ea02-e4d8-4d85-9b3d-973043279153-kube-api-access-gkrtz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:52.134719 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:52.134639 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:52.509212 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:52.509174 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz"] Oct 31 04:34:52.515914 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:34:52.515881 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2006ea02_e4d8_4d85_9b3d_973043279153.slice/crio-d8788c29367341bfd80fb7cf5ee697b4670799f8b16cc33ae6a617c670e731ca WatchSource:0}: Error finding container d8788c29367341bfd80fb7cf5ee697b4670799f8b16cc33ae6a617c670e731ca: Status 404 returned error can't find the container with id d8788c29367341bfd80fb7cf5ee697b4670799f8b16cc33ae6a617c670e731ca Oct 31 04:34:53.320658 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:53.320615 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" event={"ID":"2006ea02-e4d8-4d85-9b3d-973043279153","Type":"ContainerStarted","Data":"2f9f3f170be607660431be731dd5dcc4932cba1e0b7625b4201c8d94ad6f9f4a"} Oct 31 04:34:53.321259 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:53.320674 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" event={"ID":"2006ea02-e4d8-4d85-9b3d-973043279153","Type":"ContainerStarted","Data":"d8788c29367341bfd80fb7cf5ee697b4670799f8b16cc33ae6a617c670e731ca"} Oct 31 04:34:53.433887 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:53.433832 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" podStartSLOduration=2.433817506 podStartE2EDuration="2.433817506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:53.431634403 +0000 UTC m=+3990.432259630" watchObservedRunningTime="2026-10-31 04:34:53.433817506 +0000 UTC m=+3990.434442735" Oct 31 04:34:54.328662 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:54.328623 4426 generic.go:356] "Generic (PLEG): container finished" podID="2006ea02-e4d8-4d85-9b3d-973043279153" containerID="2f9f3f170be607660431be731dd5dcc4932cba1e0b7625b4201c8d94ad6f9f4a" exitCode=0 Oct 31 04:34:54.329142 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:54.328675 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" event={"ID":"2006ea02-e4d8-4d85-9b3d-973043279153","Type":"ContainerDied","Data":"2f9f3f170be607660431be731dd5dcc4932cba1e0b7625b4201c8d94ad6f9f4a"} Oct 31 04:34:55.466648 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:55.466616 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:55.489561 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:55.489525 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-openshift-ca\") pod \"2006ea02-e4d8-4d85-9b3d-973043279153\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " Oct 31 04:34:55.489734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:55.489622 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-xks-ca\") pod \"2006ea02-e4d8-4d85-9b3d-973043279153\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " Oct 31 04:34:55.489734 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:55.489686 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2006ea02-e4d8-4d85-9b3d-973043279153-kube-api-access-gkrtz\" (UniqueName: \"kubernetes.io/projected/2006ea02-e4d8-4d85-9b3d-973043279153-kube-api-access-gkrtz\") pod \"2006ea02-e4d8-4d85-9b3d-973043279153\" (UID: \"2006ea02-e4d8-4d85-9b3d-973043279153\") " Oct 31 04:34:55.489926 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:55.489901 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-openshift-ca" pod "2006ea02-e4d8-4d85-9b3d-973043279153" (UID: "2006ea02-e4d8-4d85-9b3d-973043279153"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:34:55.490069 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:55.490055 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:34:55.491823 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:55.491798 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-xks-ca" pod "2006ea02-e4d8-4d85-9b3d-973043279153" (UID: "2006ea02-e4d8-4d85-9b3d-973043279153"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:34:55.491969 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:55.491898 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2006ea02-e4d8-4d85-9b3d-973043279153-kube-api-access-gkrtz" pod "2006ea02-e4d8-4d85-9b3d-973043279153" (UID: "2006ea02-e4d8-4d85-9b3d-973043279153"). InnerVolumeSpecName "kube-api-access-gkrtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:34:55.591497 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:55.591418 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2006ea02-e4d8-4d85-9b3d-973043279153-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:34:55.591497 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:55.591449 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkrtz\" (UniqueName: \"kubernetes.io/projected/2006ea02-e4d8-4d85-9b3d-973043279153-kube-api-access-gkrtz\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:34:56.342412 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:56.342378 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" Oct 31 04:34:56.342412 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:56.342403 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz" event={"ID":"2006ea02-e4d8-4d85-9b3d-973043279153","Type":"ContainerDied","Data":"d8788c29367341bfd80fb7cf5ee697b4670799f8b16cc33ae6a617c670e731ca"} Oct 31 04:34:56.342648 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:34:56.342457 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8788c29367341bfd80fb7cf5ee697b4670799f8b16cc33ae6a617c670e731ca" Oct 31 04:35:02.845956 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:35:02.845917 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:35:03.257573 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:03.257529 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs"] Oct 31 04:35:03.355270 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:03.355232 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f"] Oct 31 04:35:08.258175 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.258046 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" podUID="41758200-8479-4fbb-9dd4-379fb66153fc" containerName="payload-processing" containerID="cri-o://1b2009745fa12df88f8ea10126e932797973d869dfe20fe1fbfc5850270ba37a" gracePeriod=25 Oct 31 04:35:08.355497 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.355458 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" podUID="773db981-cc62-4f1d-af35-7d09a4551e23" containerName="payload-pre-processing" containerID="cri-o://cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848" gracePeriod=25 Oct 31 04:35:08.414850 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.414812 4426 generic.go:356] "Generic (PLEG): container finished" podID="41758200-8479-4fbb-9dd4-379fb66153fc" containerID="1b2009745fa12df88f8ea10126e932797973d869dfe20fe1fbfc5850270ba37a" exitCode=0 Oct 31 04:35:08.415036 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.414889 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" event={"ID":"41758200-8479-4fbb-9dd4-379fb66153fc","Type":"ContainerDied","Data":"1b2009745fa12df88f8ea10126e932797973d869dfe20fe1fbfc5850270ba37a"} Oct 31 04:35:08.523693 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.523621 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:35:08.617985 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.617957 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:35:08.623982 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.623954 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41758200-8479-4fbb-9dd4-379fb66153fc-kube-api-access-799x9\" (UniqueName: \"kubernetes.io/projected/41758200-8479-4fbb-9dd4-379fb66153fc-kube-api-access-799x9\") pod \"41758200-8479-4fbb-9dd4-379fb66153fc\" (UID: \"41758200-8479-4fbb-9dd4-379fb66153fc\") " Oct 31 04:35:08.624159 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.624046 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41758200-8479-4fbb-9dd4-379fb66153fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41758200-8479-4fbb-9dd4-379fb66153fc-plugins-config-volume\") pod \"41758200-8479-4fbb-9dd4-379fb66153fc\" (UID: \"41758200-8479-4fbb-9dd4-379fb66153fc\") " Oct 31 04:35:08.624536 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.624508 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41758200-8479-4fbb-9dd4-379fb66153fc-plugins-config-volume" pod "41758200-8479-4fbb-9dd4-379fb66153fc" (UID: "41758200-8479-4fbb-9dd4-379fb66153fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:35:08.626155 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.626134 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41758200-8479-4fbb-9dd4-379fb66153fc-kube-api-access-799x9" pod "41758200-8479-4fbb-9dd4-379fb66153fc" (UID: "41758200-8479-4fbb-9dd4-379fb66153fc"). InnerVolumeSpecName "kube-api-access-799x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:35:08.725032 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.724995 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/773db981-cc62-4f1d-af35-7d09a4551e23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773db981-cc62-4f1d-af35-7d09a4551e23-plugins-config-volume\") pod \"773db981-cc62-4f1d-af35-7d09a4551e23\" (UID: \"773db981-cc62-4f1d-af35-7d09a4551e23\") " Oct 31 04:35:08.725282 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.725085 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773db981-cc62-4f1d-af35-7d09a4551e23-kube-api-access-7vsb8\" (UniqueName: \"kubernetes.io/projected/773db981-cc62-4f1d-af35-7d09a4551e23-kube-api-access-7vsb8\") pod \"773db981-cc62-4f1d-af35-7d09a4551e23\" (UID: \"773db981-cc62-4f1d-af35-7d09a4551e23\") " Oct 31 04:35:08.725368 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.725357 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41758200-8479-4fbb-9dd4-379fb66153fc-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:35:08.725436 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.725371 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-799x9\" (UniqueName: \"kubernetes.io/projected/41758200-8479-4fbb-9dd4-379fb66153fc-kube-api-access-799x9\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:35:08.725531 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.725504 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773db981-cc62-4f1d-af35-7d09a4551e23-plugins-config-volume" pod "773db981-cc62-4f1d-af35-7d09a4551e23" (UID: "773db981-cc62-4f1d-af35-7d09a4551e23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:35:08.727203 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.727179 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773db981-cc62-4f1d-af35-7d09a4551e23-kube-api-access-7vsb8" pod "773db981-cc62-4f1d-af35-7d09a4551e23" (UID: "773db981-cc62-4f1d-af35-7d09a4551e23"). InnerVolumeSpecName "kube-api-access-7vsb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:35:08.826023 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.825934 4426 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773db981-cc62-4f1d-af35-7d09a4551e23-plugins-config-volume\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:35:08.826023 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:08.825974 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vsb8\" (UniqueName: \"kubernetes.io/projected/773db981-cc62-4f1d-af35-7d09a4551e23-kube-api-access-7vsb8\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:35:09.424216 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.424085 4426 generic.go:356] "Generic (PLEG): container finished" podID="773db981-cc62-4f1d-af35-7d09a4551e23" containerID="cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848" exitCode=0 Oct 31 04:35:09.424216 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.424168 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" Oct 31 04:35:09.424216 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.424178 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" event={"ID":"773db981-cc62-4f1d-af35-7d09a4551e23","Type":"ContainerDied","Data":"cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848"} Oct 31 04:35:09.424834 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.424275 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f" event={"ID":"773db981-cc62-4f1d-af35-7d09a4551e23","Type":"ContainerDied","Data":"7f38b4934340918446499c20339a105ff0069d06b3b08874acff713eac609f82"} Oct 31 04:35:09.424834 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.424308 4426 scope.go:122] "RemoveContainer" containerID="cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848" Oct 31 04:35:09.426194 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.426161 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" event={"ID":"41758200-8479-4fbb-9dd4-379fb66153fc","Type":"ContainerDied","Data":"5d62a4fac841e18a86d9ebcaf9d728097b3ae198db55a1637c6ee19bc6e4b944"} Oct 31 04:35:09.426317 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.426194 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs" Oct 31 04:35:09.434708 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.434680 4426 scope.go:122] "RemoveContainer" containerID="cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848" Oct 31 04:35:09.435101 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:35:09.435078 4426 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848\": container with ID starting with cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848 not found: ID does not exist" containerID="cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848" Oct 31 04:35:09.435195 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.435110 4426 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848"} err="failed to get container status \"cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848\": rpc error: code = NotFound desc = could not find container \"cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848\": container with ID starting with cd0b3fb02c5ed1bf2412f307c5275e9f6c9824be23b1c15a562e922f62cf0848 not found: ID does not exist" Oct 31 04:35:09.435195 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.435144 4426 scope.go:122] "RemoveContainer" containerID="1b2009745fa12df88f8ea10126e932797973d869dfe20fe1fbfc5850270ba37a" Oct 31 04:35:09.579364 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.579300 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs"] Oct 31 04:35:09.583584 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.583555 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bb6bc8-86754c7dd9-mtlcs"] Oct 31 04:35:09.685671 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.685579 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f"] Oct 31 04:35:09.690092 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:09.690060 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bb6bc8-75746fdf5-4fw8f"] Oct 31 04:35:11.129496 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:11.129451 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41758200-8479-4fbb-9dd4-379fb66153fc" path="/var/lib/kubelet/pods/41758200-8479-4fbb-9dd4-379fb66153fc/volumes" Oct 31 04:35:11.130062 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:11.130040 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="773db981-cc62-4f1d-af35-7d09a4551e23" path="/var/lib/kubelet/pods/773db981-cc62-4f1d-af35-7d09a4551e23/volumes" Oct 31 04:35:15.603789 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:15.603753 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz"] Oct 31 04:35:15.650062 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:15.650025 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bb6bc8-vxbkz"] Oct 31 04:35:17.129413 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:17.129376 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2006ea02-e4d8-4d85-9b3d-973043279153" path="/var/lib/kubelet/pods/2006ea02-e4d8-4d85-9b3d-973043279153/volumes" Oct 31 04:35:19.545197 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:19.545158 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:35:22.045687 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:22.045653 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64"] Oct 31 04:35:22.091988 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:22.091956 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bft64"] Oct 31 04:35:23.128602 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:23.128563 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2701903-0553-4b83-b607-291c2c5e15b1" path="/var/lib/kubelet/pods/a2701903-0553-4b83-b607-291c2c5e15b1/volumes" Oct 31 04:35:30.933211 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:30.933170 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:35:32.914910 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:35:32.914865 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:35:42.128355 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:42.128319 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:35:50.545572 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.545529 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb"] Oct 31 04:35:50.546081 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.546024 4426 cpu_manager.go:414] "containerMap: removing container" podUID="41758200-8479-4fbb-9dd4-379fb66153fc" containerName="payload-processing" Oct 31 04:35:50.546081 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.546040 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41758200-8479-4fbb-9dd4-379fb66153fc" containerName="payload-processing" Oct 31 04:35:50.546081 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.546055 4426 cpu_manager.go:414] "containerMap: removing container" podUID="2006ea02-e4d8-4d85-9b3d-973043279153" containerName="revoke-keys" Oct 31 04:35:50.546081 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.546061 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2006ea02-e4d8-4d85-9b3d-973043279153" containerName="revoke-keys" Oct 31 04:35:50.546081 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.546080 4426 cpu_manager.go:414] "containerMap: removing container" podUID="773db981-cc62-4f1d-af35-7d09a4551e23" containerName="payload-pre-processing" Oct 31 04:35:50.546322 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.546085 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773db981-cc62-4f1d-af35-7d09a4551e23" containerName="payload-pre-processing" Oct 31 04:35:50.546322 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.546169 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="773db981-cc62-4f1d-af35-7d09a4551e23" containerName="payload-pre-processing" Oct 31 04:35:50.546322 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.546182 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="41758200-8479-4fbb-9dd4-379fb66153fc" containerName="payload-processing" Oct 31 04:35:50.546322 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.546189 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="2006ea02-e4d8-4d85-9b3d-973043279153" containerName="revoke-keys" Oct 31 04:35:50.549615 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.549596 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:50.648218 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.648172 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgjbl\"" Oct 31 04:35:50.648419 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.648232 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:35:50.648419 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.648243 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:35:50.648419 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.648256 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:35:50.651131 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.651097 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb"] Oct 31 04:35:50.656814 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.656785 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-hdwvb\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:50.656876 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.656846 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-hdwvb\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:50.656953 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.656939 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxfht\" (UniqueName: \"kubernetes.io/projected/6be4122d-8478-40e5-a977-665d7c7025c9-kube-api-access-mxfht\") pod \"maas-api-revoke-sub-f03629db-hdwvb\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:50.758539 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.758498 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-hdwvb\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:50.758730 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.758554 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxfht\" (UniqueName: \"kubernetes.io/projected/6be4122d-8478-40e5-a977-665d7c7025c9-kube-api-access-mxfht\") pod \"maas-api-revoke-sub-f03629db-hdwvb\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:50.758730 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.758621 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-hdwvb\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:50.759384 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.759352 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-hdwvb\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:50.761149 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.761098 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-hdwvb\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:50.816812 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.816732 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxfht\" (UniqueName: \"kubernetes.io/projected/6be4122d-8478-40e5-a977-665d7c7025c9-kube-api-access-mxfht\") pod \"maas-api-revoke-sub-f03629db-hdwvb\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:50.862682 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:50.862645 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:51.235342 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:51.235299 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb"] Oct 31 04:35:51.242085 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:35:51.242017 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be4122d_8478_40e5_a977_665d7c7025c9.slice/crio-5aec86514150991c258f21394411f21061d8c40f3da02919b295b952df454f30 WatchSource:0}: Error finding container 5aec86514150991c258f21394411f21061d8c40f3da02919b295b952df454f30: Status 404 returned error can't find the container with id 5aec86514150991c258f21394411f21061d8c40f3da02919b295b952df454f30 Oct 31 04:35:51.724036 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:51.723994 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" event={"ID":"6be4122d-8478-40e5-a977-665d7c7025c9","Type":"ContainerStarted","Data":"2d8b20336cdc656a22e8bef23ab2dd84dac78833a876db4f3d5491a3550c020d"} Oct 31 04:35:51.724554 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:51.724062 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" event={"ID":"6be4122d-8478-40e5-a977-665d7c7025c9","Type":"ContainerStarted","Data":"5aec86514150991c258f21394411f21061d8c40f3da02919b295b952df454f30"} Oct 31 04:35:51.834524 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:51.834404 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" podStartSLOduration=1.834387887 podStartE2EDuration="1.834387887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:35:51.832302957 +0000 UTC m=+4048.832928185" watchObservedRunningTime="2026-10-31 04:35:51.834387887 +0000 UTC m=+4048.835013115" Oct 31 04:35:52.737334 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:52.737294 4426 generic.go:356] "Generic (PLEG): container finished" podID="6be4122d-8478-40e5-a977-665d7c7025c9" containerID="2d8b20336cdc656a22e8bef23ab2dd84dac78833a876db4f3d5491a3550c020d" exitCode=0 Oct 31 04:35:52.737788 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:52.737345 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" event={"ID":"6be4122d-8478-40e5-a977-665d7c7025c9","Type":"ContainerDied","Data":"2d8b20336cdc656a22e8bef23ab2dd84dac78833a876db4f3d5491a3550c020d"} Oct 31 04:35:53.883495 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:53.883462 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:53.989242 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:53.989210 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-xks-ca\") pod \"6be4122d-8478-40e5-a977-665d7c7025c9\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " Oct 31 04:35:53.989424 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:53.989285 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be4122d-8478-40e5-a977-665d7c7025c9-kube-api-access-mxfht\" (UniqueName: \"kubernetes.io/projected/6be4122d-8478-40e5-a977-665d7c7025c9-kube-api-access-mxfht\") pod \"6be4122d-8478-40e5-a977-665d7c7025c9\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " Oct 31 04:35:53.989469 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:53.989444 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-openshift-ca\") pod \"6be4122d-8478-40e5-a977-665d7c7025c9\" (UID: \"6be4122d-8478-40e5-a977-665d7c7025c9\") " Oct 31 04:35:53.989834 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:53.989808 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-openshift-ca" pod "6be4122d-8478-40e5-a977-665d7c7025c9" (UID: "6be4122d-8478-40e5-a977-665d7c7025c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:35:53.991317 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:53.991287 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-xks-ca" pod "6be4122d-8478-40e5-a977-665d7c7025c9" (UID: "6be4122d-8478-40e5-a977-665d7c7025c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:35:53.991437 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:53.991394 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be4122d-8478-40e5-a977-665d7c7025c9-kube-api-access-mxfht" pod "6be4122d-8478-40e5-a977-665d7c7025c9" (UID: "6be4122d-8478-40e5-a977-665d7c7025c9"). InnerVolumeSpecName "kube-api-access-mxfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:35:54.091024 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:54.090984 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:35:54.091024 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:54.091016 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxfht\" (UniqueName: \"kubernetes.io/projected/6be4122d-8478-40e5-a977-665d7c7025c9-kube-api-access-mxfht\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:35:54.091024 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:54.091030 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be4122d-8478-40e5-a977-665d7c7025c9-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:35:54.751705 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:54.751674 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" Oct 31 04:35:54.751907 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:54.751715 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb" event={"ID":"6be4122d-8478-40e5-a977-665d7c7025c9","Type":"ContainerDied","Data":"5aec86514150991c258f21394411f21061d8c40f3da02919b295b952df454f30"} Oct 31 04:35:54.751907 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:35:54.751765 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aec86514150991c258f21394411f21061d8c40f3da02919b295b952df454f30" Oct 31 04:36:02.848803 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:36:02.848762 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:36:03.209873 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:03.209835 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:36:11.382866 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.382829 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882"] Oct 31 04:36:11.383321 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.383307 4426 cpu_manager.go:414] "containerMap: removing container" podUID="6be4122d-8478-40e5-a977-665d7c7025c9" containerName="revoke-keys" Oct 31 04:36:11.383392 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.383322 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be4122d-8478-40e5-a977-665d7c7025c9" containerName="revoke-keys" Oct 31 04:36:11.383461 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.383400 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be4122d-8478-40e5-a977-665d7c7025c9" containerName="revoke-keys" Oct 31 04:36:11.386788 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.386758 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:11.438496 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.438451 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:36:11.485018 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.484987 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgjbl\"" Oct 31 04:36:11.485304 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.485281 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:36:11.485367 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.485316 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:36:11.488136 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.488102 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882"] Oct 31 04:36:11.501168 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.501114 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-jd882\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:11.501348 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.501183 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrpt\" (UniqueName: \"kubernetes.io/projected/dd0b7fde-ea48-4c61-96be-50817e2febb5-kube-api-access-znrpt\") pod \"maas-api-revoke-sub-0165d7d2-jd882\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:11.501348 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.501270 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-jd882\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:11.602530 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.602494 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-jd882\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:11.602530 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.602536 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrpt\" (UniqueName: \"kubernetes.io/projected/dd0b7fde-ea48-4c61-96be-50817e2febb5-kube-api-access-znrpt\") pod \"maas-api-revoke-sub-0165d7d2-jd882\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:11.602794 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.602565 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-jd882\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:11.603216 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.603190 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-jd882\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:11.604995 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.604969 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-jd882\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:11.660167 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.660115 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrpt\" (UniqueName: \"kubernetes.io/projected/dd0b7fde-ea48-4c61-96be-50817e2febb5-kube-api-access-znrpt\") pod \"maas-api-revoke-sub-0165d7d2-jd882\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:11.700990 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:11.700954 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:12.100461 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:12.100358 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882"] Oct 31 04:36:12.106725 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:36:12.106691 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0b7fde_ea48_4c61_96be_50817e2febb5.slice/crio-8a0719f5ef69ab70ee429a36ed9106f776a5dd29a3adbe895f21803c78f0a8f1 WatchSource:0}: Error finding container 8a0719f5ef69ab70ee429a36ed9106f776a5dd29a3adbe895f21803c78f0a8f1: Status 404 returned error can't find the container with id 8a0719f5ef69ab70ee429a36ed9106f776a5dd29a3adbe895f21803c78f0a8f1 Oct 31 04:36:12.879671 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:12.879571 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" event={"ID":"dd0b7fde-ea48-4c61-96be-50817e2febb5","Type":"ContainerStarted","Data":"31d773fd7c788324d72a8e052fc7f189b407bae9fa74250a4e877a9b67903234"} Oct 31 04:36:12.879671 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:12.879627 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" event={"ID":"dd0b7fde-ea48-4c61-96be-50817e2febb5","Type":"ContainerStarted","Data":"8a0719f5ef69ab70ee429a36ed9106f776a5dd29a3adbe895f21803c78f0a8f1"} Oct 31 04:36:12.989174 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:12.989092 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" podStartSLOduration=1.9890771250000001 podStartE2EDuration="1.989077125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:36:12.987894366 +0000 UTC m=+4069.988519594" watchObservedRunningTime="2026-10-31 04:36:12.989077125 +0000 UTC m=+4069.989702353" Oct 31 04:36:13.886566 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:13.886529 4426 generic.go:356] "Generic (PLEG): container finished" podID="dd0b7fde-ea48-4c61-96be-50817e2febb5" containerID="31d773fd7c788324d72a8e052fc7f189b407bae9fa74250a4e877a9b67903234" exitCode=0 Oct 31 04:36:13.887022 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:13.886617 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" event={"ID":"dd0b7fde-ea48-4c61-96be-50817e2febb5","Type":"ContainerDied","Data":"31d773fd7c788324d72a8e052fc7f189b407bae9fa74250a4e877a9b67903234"} Oct 31 04:36:15.024206 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.024179 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:15.036373 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.036349 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-openshift-ca\") pod \"dd0b7fde-ea48-4c61-96be-50817e2febb5\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " Oct 31 04:36:15.036485 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.036412 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0b7fde-ea48-4c61-96be-50817e2febb5-kube-api-access-znrpt\" (UniqueName: \"kubernetes.io/projected/dd0b7fde-ea48-4c61-96be-50817e2febb5-kube-api-access-znrpt\") pod \"dd0b7fde-ea48-4c61-96be-50817e2febb5\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " Oct 31 04:36:15.036675 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.036651 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-openshift-ca" pod "dd0b7fde-ea48-4c61-96be-50817e2febb5" (UID: "dd0b7fde-ea48-4c61-96be-50817e2febb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:36:15.038433 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.038409 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0b7fde-ea48-4c61-96be-50817e2febb5-kube-api-access-znrpt" pod "dd0b7fde-ea48-4c61-96be-50817e2febb5" (UID: "dd0b7fde-ea48-4c61-96be-50817e2febb5"). InnerVolumeSpecName "kube-api-access-znrpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:36:15.136841 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.136809 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-xks-ca\") pod \"dd0b7fde-ea48-4c61-96be-50817e2febb5\" (UID: \"dd0b7fde-ea48-4c61-96be-50817e2febb5\") " Oct 31 04:36:15.137109 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.137088 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znrpt\" (UniqueName: \"kubernetes.io/projected/dd0b7fde-ea48-4c61-96be-50817e2febb5-kube-api-access-znrpt\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:36:15.137207 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.137110 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:36:15.139274 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.139243 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-xks-ca" pod "dd0b7fde-ea48-4c61-96be-50817e2febb5" (UID: "dd0b7fde-ea48-4c61-96be-50817e2febb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:36:15.238237 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.238200 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0b7fde-ea48-4c61-96be-50817e2febb5-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:36:15.907283 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.907236 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" event={"ID":"dd0b7fde-ea48-4c61-96be-50817e2febb5","Type":"ContainerDied","Data":"8a0719f5ef69ab70ee429a36ed9106f776a5dd29a3adbe895f21803c78f0a8f1"} Oct 31 04:36:15.907468 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.907315 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a0719f5ef69ab70ee429a36ed9106f776a5dd29a3adbe895f21803c78f0a8f1" Oct 31 04:36:15.907468 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:15.907256 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882" Oct 31 04:36:23.021927 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.021889 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g"] Oct 31 04:36:23.022523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.022466 4426 cpu_manager.go:414] "containerMap: removing container" podUID="dd0b7fde-ea48-4c61-96be-50817e2febb5" containerName="revoke-keys" Oct 31 04:36:23.022523 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.022482 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0b7fde-ea48-4c61-96be-50817e2febb5" containerName="revoke-keys" Oct 31 04:36:23.022613 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.022549 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0b7fde-ea48-4c61-96be-50817e2febb5" containerName="revoke-keys" Oct 31 04:36:23.026092 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.026071 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:23.077210 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.077178 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:36:23.113197 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.113160 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jk46g\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:23.113396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.113313 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jk46g\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:23.113396 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.113367 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc2p6\" (UniqueName: \"kubernetes.io/projected/d181a6d5-b87e-4f8c-8774-0ed5df832c04-kube-api-access-lc2p6\") pod \"maas-api-revoke-sub-6b19338f-jk46g\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:23.123517 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.123480 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:36:23.123693 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.123574 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:36:23.123693 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.123669 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgjbl\"" Oct 31 04:36:23.129004 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.128979 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g"] Oct 31 04:36:23.214543 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.214509 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2p6\" (UniqueName: \"kubernetes.io/projected/d181a6d5-b87e-4f8c-8774-0ed5df832c04-kube-api-access-lc2p6\") pod \"maas-api-revoke-sub-6b19338f-jk46g\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:23.214753 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.214585 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jk46g\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:23.214753 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.214678 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jk46g\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:23.215401 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.215374 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jk46g\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:23.262588 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.262556 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:36:23.267815 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.267793 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jk46g\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:23.271779 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.271755 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2p6\" (UniqueName: \"kubernetes.io/projected/d181a6d5-b87e-4f8c-8774-0ed5df832c04-kube-api-access-lc2p6\") pod \"maas-api-revoke-sub-6b19338f-jk46g\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:23.339985 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.339896 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:23.711548 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.711513 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g"] Oct 31 04:36:23.718578 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:36:23.718537 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd181a6d5_b87e_4f8c_8774_0ed5df832c04.slice/crio-6382c915fd157e8fc65373a507a423e53483a0b68f62a06c486f2aabea638e9d WatchSource:0}: Error finding container 6382c915fd157e8fc65373a507a423e53483a0b68f62a06c486f2aabea638e9d: Status 404 returned error can't find the container with id 6382c915fd157e8fc65373a507a423e53483a0b68f62a06c486f2aabea638e9d Oct 31 04:36:23.959019 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:23.958975 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" event={"ID":"d181a6d5-b87e-4f8c-8774-0ed5df832c04","Type":"ContainerStarted","Data":"6382c915fd157e8fc65373a507a423e53483a0b68f62a06c486f2aabea638e9d"} Oct 31 04:36:24.966840 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:24.966802 4426 generic.go:356] "Generic (PLEG): container finished" podID="d181a6d5-b87e-4f8c-8774-0ed5df832c04" containerID="74f254d0d4c8bfa007b8bef4ea58a1526bb1ba3619725a89daf2208e96d70de0" exitCode=0 Oct 31 04:36:24.967276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:24.966885 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" event={"ID":"d181a6d5-b87e-4f8c-8774-0ed5df832c04","Type":"ContainerDied","Data":"74f254d0d4c8bfa007b8bef4ea58a1526bb1ba3619725a89daf2208e96d70de0"} Oct 31 04:36:26.112103 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.112075 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:26.142033 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.142004 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-xks-ca\") pod \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " Oct 31 04:36:26.142236 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.142083 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-openshift-ca\") pod \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " Oct 31 04:36:26.142236 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.142187 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d181a6d5-b87e-4f8c-8774-0ed5df832c04-kube-api-access-lc2p6\" (UniqueName: \"kubernetes.io/projected/d181a6d5-b87e-4f8c-8774-0ed5df832c04-kube-api-access-lc2p6\") pod \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\" (UID: \"d181a6d5-b87e-4f8c-8774-0ed5df832c04\") " Oct 31 04:36:26.142673 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.142505 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-openshift-ca" pod "d181a6d5-b87e-4f8c-8774-0ed5df832c04" (UID: "d181a6d5-b87e-4f8c-8774-0ed5df832c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:36:26.144340 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.144318 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-xks-ca" pod "d181a6d5-b87e-4f8c-8774-0ed5df832c04" (UID: "d181a6d5-b87e-4f8c-8774-0ed5df832c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:36:26.144426 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.144393 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d181a6d5-b87e-4f8c-8774-0ed5df832c04-kube-api-access-lc2p6" pod "d181a6d5-b87e-4f8c-8774-0ed5df832c04" (UID: "d181a6d5-b87e-4f8c-8774-0ed5df832c04"). InnerVolumeSpecName "kube-api-access-lc2p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:36:26.244218 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.244094 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-openshift-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:36:26.244218 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.244166 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc2p6\" (UniqueName: \"kubernetes.io/projected/d181a6d5-b87e-4f8c-8774-0ed5df832c04-kube-api-access-lc2p6\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:36:26.244218 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.244187 4426 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d181a6d5-b87e-4f8c-8774-0ed5df832c04-maas-api-xks-ca\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:36:26.980007 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.979975 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" Oct 31 04:36:26.980252 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.980010 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g" event={"ID":"d181a6d5-b87e-4f8c-8774-0ed5df832c04","Type":"ContainerDied","Data":"6382c915fd157e8fc65373a507a423e53483a0b68f62a06c486f2aabea638e9d"} Oct 31 04:36:26.980252 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:26.980086 4426 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6382c915fd157e8fc65373a507a423e53483a0b68f62a06c486f2aabea638e9d" Oct 31 04:36:27.193209 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:27.193175 4426 scope.go:122] "RemoveContainer" containerID="3eea4a39bd399417b1570ab28cce133b7040debdb1f741c95c4cac3e9c75dd49" Oct 31 04:36:32.845799 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:36:32.845761 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:36:35.518273 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:35.518239 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66fb56dd6d-wxqfb_2e9f99a4-7d1c-4783-9c5f-d565dd5f0931/authorino/0.log" Oct 31 04:36:35.726623 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:35.726541 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-22j29"] Oct 31 04:36:56.307668 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:56.307629 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-57d9f499dc-7th7b_41eb1d0e-8fed-41d8-99a1-91c359891566/manager/0.log" Oct 31 04:36:56.685213 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:56.685182 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n4sbl_c284e318-4c88-4554-bcf3-4ce3da29357a/manager/0.log" Oct 31 04:36:57.964068 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:57.964038 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-565fd9c896-fjbt8_47b46629-173e-4e33-b7a0-62236a5d7251/manager/0.log" Oct 31 04:36:58.458365 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:58.458332 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-c8lvn_f6dda95c-c95e-4951-aa12-6809320e9350/server/0.log" Oct 31 04:36:58.686632 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:58.686599 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2lsx8_fa07d775-b0fd-4441-b8ff-b639991d326e/manager/0.log" Oct 31 04:36:58.940853 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:36:58.940829 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-l7zq6_c14f5021-0344-46ff-a0ad-85b011c7ade3/proxy/0.log" Oct 31 04:37:00.369283 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:00.369250 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-jd882_dd0b7fde-ea48-4c61-96be-50817e2febb5/revoke-keys/0.log" Oct 31 04:37:00.601962 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:00.601931 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-jk46g_d181a6d5-b87e-4f8c-8774-0ed5df832c04/revoke-keys/0.log" Oct 31 04:37:00.828263 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:00.828233 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-hdwvb_6be4122d-8478-40e5-a977-665d7c7025c9/revoke-keys/0.log" Oct 31 04:37:02.573949 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:02.573914 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66fb56dd6d-wxqfb_2e9f99a4-7d1c-4783-9c5f-d565dd5f0931/authorino/0.log" Oct 31 04:37:02.845577 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:37:02.845490 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:37:03.047979 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:03.047945 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gc4kf_c79d294e-f5fc-4f02-bdda-5aa7885145bf/manager/0.log" Oct 31 04:37:03.287578 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:03.287543 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8gg9j_60f7506b-536a-4604-952d-d7447f9a3787/manager/0.log" Oct 31 04:37:03.745765 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:03.745728 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7pknt_471df0d0-0980-416c-a410-dabeaeaa01ee/manager/0.log" Oct 31 04:37:04.065137 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:04.065047 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-22j29_155f33ed-b4f9-4914-934d-82b04dcba3a1/limitador/0.log" Oct 31 04:37:04.299759 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:04.299727 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2dljn_7a80e800-84ce-434c-98f6-f0e8823ed5d7/manager/0.log" Oct 31 04:37:05.108960 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:05.108921 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9_16b2427e-bf1e-4892-ad55-9d5f2b93849b/istio-proxy/0.log" Oct 31 04:37:16.654772 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:16.654734 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-fp9bz_4d600e53-d628-4f89-b6c6-186d3432e300/istio-proxy/0.log" Oct 31 04:37:16.928740 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:16.928661 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-fxl7k_76a0c679-9315-4024-a425-ae42d2716f0c/istio-proxy/0.log" Oct 31 04:37:18.279115 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:18.279081 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-tg5t2_8b99c850-aaaf-4055-9a35-dc3ce08085d7/istio-proxy/0.log" Oct 31 04:37:18.830764 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:18.830728 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht_f5f118b0-a36e-4372-8162-8e9e19d7441b/istio-proxy/0.log" Oct 31 04:37:19.100363 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:19.100288 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9wmqv_a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f/payload-pre-processing/0.log" Oct 31 04:37:29.322761 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:29.322731 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-97mff_e44d8d56-e743-4335-a287-2330fdeeb3a1/payload-processing/0.log" Oct 31 04:37:32.846297 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:37:32.846253 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:37:40.520663 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:40.520627 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh_c785f917-ad62-4bcf-9b30-dcbebdbda03b/main/0.log" Oct 31 04:37:41.278637 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:41.278606 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-bc65b_71b3bae1-fa55-4a39-873b-69c1da86ecf5/main/0.log" Oct 31 04:37:41.806376 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:41.806345 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qlm9m_33c358fc-a3bb-4db1-80b3-d1f799e29621/main/0.log" Oct 31 04:37:43.326678 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:37:43.326647 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66fb56dd6d-wxqfb_2e9f99a4-7d1c-4783-9c5f-d565dd5f0931/authorino/0.log" Oct 31 04:38:02.845551 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:38:02.845499 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:38:13.272585 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:13.272503 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-57d9f499dc-7th7b_41eb1d0e-8fed-41d8-99a1-91c359891566/manager/0.log" Oct 31 04:38:13.647300 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:13.647274 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n4sbl_c284e318-4c88-4554-bcf3-4ce3da29357a/manager/0.log" Oct 31 04:38:14.699849 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:14.699813 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-565fd9c896-fjbt8_47b46629-173e-4e33-b7a0-62236a5d7251/manager/0.log" Oct 31 04:38:15.147542 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:15.147517 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-c8lvn_f6dda95c-c95e-4951-aa12-6809320e9350/server/0.log" Oct 31 04:38:15.392754 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:15.392723 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2lsx8_fa07d775-b0fd-4441-b8ff-b639991d326e/manager/0.log" Oct 31 04:38:15.661884 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:15.661843 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-l7zq6_c14f5021-0344-46ff-a0ad-85b011c7ade3/proxy/0.log" Oct 31 04:38:16.942783 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:16.942744 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-jd882_dd0b7fde-ea48-4c61-96be-50817e2febb5/revoke-keys/0.log" Oct 31 04:38:17.180678 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:17.180648 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-jk46g_d181a6d5-b87e-4f8c-8774-0ed5df832c04/revoke-keys/0.log" Oct 31 04:38:17.407339 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:17.407310 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-hdwvb_6be4122d-8478-40e5-a977-665d7c7025c9/revoke-keys/0.log" Oct 31 04:38:23.928090 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:23.928056 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:38:23.936491 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:23.936464 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awqlgc_ccff26f6-b677-45ee-920a-1692c44f4fe9/extract/0.log" Oct 31 04:38:27.224684 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:27.224649 4426 scope.go:122] "RemoveContainer" containerID="2fd15e75fb52f932dd703afe5c5bbc79205e5f211e3c9cec306cbad6372e15db" Oct 31 04:38:28.852859 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:28.852823 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66fb56dd6d-wxqfb_2e9f99a4-7d1c-4783-9c5f-d565dd5f0931/authorino/0.log" Oct 31 04:38:29.347758 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:29.347723 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gc4kf_c79d294e-f5fc-4f02-bdda-5aa7885145bf/manager/0.log" Oct 31 04:38:32.844765 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:38:32.844723 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:38:39.596273 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:39.596239 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8gg9j_60f7506b-536a-4604-952d-d7447f9a3787/manager/0.log" Oct 31 04:38:40.084808 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:40.084777 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7pknt_471df0d0-0980-416c-a410-dabeaeaa01ee/manager/0.log" Oct 31 04:38:40.576356 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:40.576327 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-22j29_155f33ed-b4f9-4914-934d-82b04dcba3a1/limitador/0.log" Oct 31 04:38:40.837611 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:40.837519 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2dljn_7a80e800-84ce-434c-98f6-f0e8823ed5d7/manager/0.log" Oct 31 04:38:41.464973 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:41.464932 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9_16b2427e-bf1e-4892-ad55-9d5f2b93849b/istio-proxy/0.log" Oct 31 04:38:43.087744 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:43.087714 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-fp9bz_4d600e53-d628-4f89-b6c6-186d3432e300/istio-proxy/0.log" Oct 31 04:38:43.378102 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:43.378022 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-fxl7k_76a0c679-9315-4024-a425-ae42d2716f0c/istio-proxy/0.log" Oct 31 04:38:54.824601 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:54.824567 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-tg5t2_8b99c850-aaaf-4055-9a35-dc3ce08085d7/istio-proxy/0.log" Oct 31 04:38:55.429800 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:55.429768 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6f8685f4cf-dxbht_f5f118b0-a36e-4372-8162-8e9e19d7441b/istio-proxy/0.log" Oct 31 04:38:55.663912 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:55.663883 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9wmqv_a2dcf8e8-043d-48d9-9c2e-cdb93c878b5f/payload-pre-processing/0.log" Oct 31 04:38:55.907144 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:55.907107 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-97mff_e44d8d56-e743-4335-a287-2330fdeeb3a1/payload-processing/0.log" Oct 31 04:38:56.858809 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:56.858770 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-l9kfh_c785f917-ad62-4bcf-9b30-dcbebdbda03b/main/0.log" Oct 31 04:38:57.638985 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:57.638944 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-bc65b_71b3bae1-fa55-4a39-873b-69c1da86ecf5/main/0.log" Oct 31 04:38:58.202315 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:38:58.202284 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qlm9m_33c358fc-a3bb-4db1-80b3-d1f799e29621/main/0.log" Oct 31 04:39:02.845251 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:39:02.845214 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:39:32.845901 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:39:32.845856 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:40:02.845152 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:40:02.845092 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:40:20.262619 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:40:20.262582 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-565fd9c896-fjbt8_47b46629-173e-4e33-b7a0-62236a5d7251/manager/0.log" Oct 31 04:40:20.834233 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:40:20.834199 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-565fd9c896-fjbt8_47b46629-173e-4e33-b7a0-62236a5d7251/manager/0.log" Oct 31 04:40:21.617660 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:40:21.617629 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-97mff_e44d8d56-e743-4335-a287-2330fdeeb3a1/payload-processing/0.log" Oct 31 04:40:21.841104 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:40:21.841076 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-97mff_e44d8d56-e743-4335-a287-2330fdeeb3a1/payload-processing/0.log" Oct 31 04:40:32.845828 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:40:32.845788 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:40:54.094103 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:40:54.094065 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb"] Oct 31 04:40:54.096613 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:40:54.096586 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hdwvb"] Oct 31 04:40:55.128663 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:40:55.128622 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be4122d-8478-40e5-a977-665d7c7025c9" path="/var/lib/kubelet/pods/6be4122d-8478-40e5-a977-665d7c7025c9/volumes" Oct 31 04:41:02.845601 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:41:02.845559 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:41:16.045039 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:41:16.044959 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882"] Oct 31 04:41:16.091481 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:41:16.091445 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-jd882"] Oct 31 04:41:17.128089 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:41:17.128051 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0b7fde-ea48-4c61-96be-50817e2febb5" path="/var/lib/kubelet/pods/dd0b7fde-ea48-4c61-96be-50817e2febb5/volumes" Oct 31 04:41:27.046876 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:41:27.046837 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g"] Oct 31 04:41:27.093078 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:41:27.093039 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jk46g"] Oct 31 04:41:27.127853 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:41:27.127817 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d181a6d5-b87e-4f8c-8774-0ed5df832c04" path="/var/lib/kubelet/pods/d181a6d5-b87e-4f8c-8774-0ed5df832c04/volumes" Oct 31 04:41:27.268658 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:41:27.268626 4426 scope.go:122] "RemoveContainer" containerID="2f9f3f170be607660431be731dd5dcc4932cba1e0b7625b4201c8d94ad6f9f4a" Oct 31 04:41:32.845547 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:41:32.845500 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:41:58.834121 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:41:58.834082 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k"] Oct 31 04:41:58.835381 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:41:58.834302 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" containerID="cri-o://f575c82e8a43286fa4c28acdadfa97e65467a001c7c09631a1cf6941ff73af04" gracePeriod=30 Oct 31 04:41:58.851701 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:41:58.851667 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz"] Oct 31 04:41:58.851948 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:41:58.851916 4426 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" containerName="istio-proxy" containerID="cri-o://990d7b5af68d24b3bd3c7a4f29dc0182e33e6e9c8a6b8247d043d06ef976216c" gracePeriod=30 Oct 31 04:42:02.844764 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:42:02.844720 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:42:04.036564 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.036527 4426 generic.go:356] "Generic (PLEG): container finished" podID="76a0c679-9315-4024-a425-ae42d2716f0c" containerID="f575c82e8a43286fa4c28acdadfa97e65467a001c7c09631a1cf6941ff73af04" exitCode=0 Oct 31 04:42:04.037154 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.036733 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" event={"ID":"76a0c679-9315-4024-a425-ae42d2716f0c","Type":"ContainerDied","Data":"f575c82e8a43286fa4c28acdadfa97e65467a001c7c09631a1cf6941ff73af04"} Oct 31 04:42:04.038902 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.038879 4426 generic.go:356] "Generic (PLEG): container finished" podID="4d600e53-d628-4f89-b6c6-186d3432e300" containerID="990d7b5af68d24b3bd3c7a4f29dc0182e33e6e9c8a6b8247d043d06ef976216c" exitCode=0 Oct 31 04:42:04.039018 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.038985 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" event={"ID":"4d600e53-d628-4f89-b6c6-186d3432e300","Type":"ContainerDied","Data":"990d7b5af68d24b3bd3c7a4f29dc0182e33e6e9c8a6b8247d043d06ef976216c"} Oct 31 04:42:04.097878 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.097851 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 04:42:04.133854 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.133830 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 04:42:04.163151 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163102 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-data\") pod \"4d600e53-d628-4f89-b6c6-186d3432e300\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " Oct 31 04:42:04.163311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163179 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a0c679-9315-4024-a425-ae42d2716f0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a0c679-9315-4024-a425-ae42d2716f0c-istiod-ca-cert\") pod \"76a0c679-9315-4024-a425-ae42d2716f0c\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " Oct 31 04:42:04.163311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163209 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-socket\") pod \"76a0c679-9315-4024-a425-ae42d2716f0c\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " Oct 31 04:42:04.163311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163241 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-credential-socket\") pod \"4d600e53-d628-4f89-b6c6-186d3432e300\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " Oct 31 04:42:04.163311 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163277 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d600e53-d628-4f89-b6c6-186d3432e300-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d600e53-d628-4f89-b6c6-186d3432e300-istiod-ca-cert\") pod \"4d600e53-d628-4f89-b6c6-186d3432e300\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " Oct 31 04:42:04.163538 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163326 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-certs\") pod \"4d600e53-d628-4f89-b6c6-186d3432e300\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " Oct 31 04:42:04.163538 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163386 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-data" pod "4d600e53-d628-4f89-b6c6-186d3432e300" (UID: "4d600e53-d628-4f89-b6c6-186d3432e300"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:04.163538 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163443 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-envoy\") pod \"4d600e53-d628-4f89-b6c6-186d3432e300\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " Oct 31 04:42:04.163538 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163492 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-envoy\") pod \"76a0c679-9315-4024-a425-ae42d2716f0c\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " Oct 31 04:42:04.163538 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163530 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-istio-token\" (UniqueName: \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-istio-token\") pod \"76a0c679-9315-4024-a425-ae42d2716f0c\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " Oct 31 04:42:04.163821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163544 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-credential-socket" pod "4d600e53-d628-4f89-b6c6-186d3432e300" (UID: "4d600e53-d628-4f89-b6c6-186d3432e300"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:04.163821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163544 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-socket" pod "76a0c679-9315-4024-a425-ae42d2716f0c" (UID: "76a0c679-9315-4024-a425-ae42d2716f0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:04.163821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163565 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-istio-token\" (UniqueName: \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-istio-token\") pod \"4d600e53-d628-4f89-b6c6-186d3432e300\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " Oct 31 04:42:04.163821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163618 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-kube-api-access-4fs5n\" (UniqueName: \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-kube-api-access-4fs5n\") pod \"4d600e53-d628-4f89-b6c6-186d3432e300\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " Oct 31 04:42:04.163821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163685 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-data\") pod \"76a0c679-9315-4024-a425-ae42d2716f0c\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " Oct 31 04:42:04.163821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163724 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-kube-api-access-bq54p\" (UniqueName: \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-kube-api-access-bq54p\") pod \"76a0c679-9315-4024-a425-ae42d2716f0c\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " Oct 31 04:42:04.163821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163764 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-socket\") pod \"4d600e53-d628-4f89-b6c6-186d3432e300\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " Oct 31 04:42:04.163821 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163807 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76a0c679-9315-4024-a425-ae42d2716f0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a0c679-9315-4024-a425-ae42d2716f0c-istio-podinfo\") pod \"76a0c679-9315-4024-a425-ae42d2716f0c\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " Oct 31 04:42:04.164276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163840 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d600e53-d628-4f89-b6c6-186d3432e300-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d600e53-d628-4f89-b6c6-186d3432e300-istio-podinfo\") pod \"4d600e53-d628-4f89-b6c6-186d3432e300\" (UID: \"4d600e53-d628-4f89-b6c6-186d3432e300\") " Oct 31 04:42:04.164276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163869 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-certs\") pod \"76a0c679-9315-4024-a425-ae42d2716f0c\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " Oct 31 04:42:04.164276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163618 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-certs" pod "4d600e53-d628-4f89-b6c6-186d3432e300" (UID: "4d600e53-d628-4f89-b6c6-186d3432e300"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:04.164276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163665 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d600e53-d628-4f89-b6c6-186d3432e300-istiod-ca-cert" pod "4d600e53-d628-4f89-b6c6-186d3432e300" (UID: "4d600e53-d628-4f89-b6c6-186d3432e300"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:42:04.164276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163688 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a0c679-9315-4024-a425-ae42d2716f0c-istiod-ca-cert" pod "76a0c679-9315-4024-a425-ae42d2716f0c" (UID: "76a0c679-9315-4024-a425-ae42d2716f0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:42:04.164276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.163957 4426 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-credential-socket\") pod \"76a0c679-9315-4024-a425-ae42d2716f0c\" (UID: \"76a0c679-9315-4024-a425-ae42d2716f0c\") " Oct 31 04:42:04.164276 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.164034 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-data" pod "76a0c679-9315-4024-a425-ae42d2716f0c" (UID: "76a0c679-9315-4024-a425-ae42d2716f0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:04.164689 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.164349 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.164689 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.164370 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a0c679-9315-4024-a425-ae42d2716f0c-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.164689 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.164384 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.164689 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.164399 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.164689 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.164413 4426 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d600e53-d628-4f89-b6c6-186d3432e300-istiod-ca-cert\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.164689 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.164427 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.164689 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.164443 4426 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-data\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.164689 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.164661 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-socket" pod "4d600e53-d628-4f89-b6c6-186d3432e300" (UID: "4d600e53-d628-4f89-b6c6-186d3432e300"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:04.164689 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.164685 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-credential-socket" pod "76a0c679-9315-4024-a425-ae42d2716f0c" (UID: "76a0c679-9315-4024-a425-ae42d2716f0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:04.165227 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.164728 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-certs" pod "76a0c679-9315-4024-a425-ae42d2716f0c" (UID: "76a0c679-9315-4024-a425-ae42d2716f0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:04.166283 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.166256 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-envoy" pod "4d600e53-d628-4f89-b6c6-186d3432e300" (UID: "4d600e53-d628-4f89-b6c6-186d3432e300"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:04.166408 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.166347 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-istio-token" pod "76a0c679-9315-4024-a425-ae42d2716f0c" (UID: "76a0c679-9315-4024-a425-ae42d2716f0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:42:04.166583 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.166559 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-envoy" pod "76a0c679-9315-4024-a425-ae42d2716f0c" (UID: "76a0c679-9315-4024-a425-ae42d2716f0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:04.166663 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.166612 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-kube-api-access-4fs5n" pod "4d600e53-d628-4f89-b6c6-186d3432e300" (UID: "4d600e53-d628-4f89-b6c6-186d3432e300"). InnerVolumeSpecName "kube-api-access-4fs5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:42:04.166737 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.166715 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d600e53-d628-4f89-b6c6-186d3432e300-istio-podinfo" pod "4d600e53-d628-4f89-b6c6-186d3432e300" (UID: "4d600e53-d628-4f89-b6c6-186d3432e300"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:42:04.166868 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.166851 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-istio-token" pod "4d600e53-d628-4f89-b6c6-186d3432e300" (UID: "4d600e53-d628-4f89-b6c6-186d3432e300"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:42:04.166868 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.166856 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76a0c679-9315-4024-a425-ae42d2716f0c-istio-podinfo" pod "76a0c679-9315-4024-a425-ae42d2716f0c" (UID: "76a0c679-9315-4024-a425-ae42d2716f0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:42:04.167491 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.167473 4426 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-kube-api-access-bq54p" pod "76a0c679-9315-4024-a425-ae42d2716f0c" (UID: "76a0c679-9315-4024-a425-ae42d2716f0c"). InnerVolumeSpecName "kube-api-access-bq54p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:42:04.266158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.265201 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.266158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.265237 4426 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-istio-envoy\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.266158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.265252 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.266158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.265267 4426 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-istio-token\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.266158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.265284 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fs5n\" (UniqueName: \"kubernetes.io/projected/4d600e53-d628-4f89-b6c6-186d3432e300-kube-api-access-4fs5n\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.266158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.265297 4426 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq54p\" (UniqueName: \"kubernetes.io/projected/76a0c679-9315-4024-a425-ae42d2716f0c-kube-api-access-bq54p\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.266158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.265307 4426 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d600e53-d628-4f89-b6c6-186d3432e300-workload-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.266158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.265315 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a0c679-9315-4024-a425-ae42d2716f0c-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.266158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.265324 4426 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d600e53-d628-4f89-b6c6-186d3432e300-istio-podinfo\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.266158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.265332 4426 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-workload-certs\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:04.266158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:04.265340 4426 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a0c679-9315-4024-a425-ae42d2716f0c-credential-socket\") on node \"ip-10-0-134-106.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 04:42:05.045859 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:05.045820 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" Oct 31 04:42:05.045859 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:05.045833 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k" event={"ID":"76a0c679-9315-4024-a425-ae42d2716f0c","Type":"ContainerDied","Data":"7c6fe55f91b3ae65a45fda69184bbafd88fbf051ac08dd5820e0567bbaab8328"} Oct 31 04:42:05.046414 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:05.045940 4426 scope.go:122] "RemoveContainer" containerID="f575c82e8a43286fa4c28acdadfa97e65467a001c7c09631a1cf6941ff73af04" Oct 31 04:42:05.048531 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:05.048509 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" event={"ID":"4d600e53-d628-4f89-b6c6-186d3432e300","Type":"ContainerDied","Data":"61e2ed21683b224bd594bbbd60c9a2cfee2d8656a4393441857b7ee0747f2b4d"} Oct 31 04:42:05.048631 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:05.048532 4426 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz" Oct 31 04:42:05.058675 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:05.058604 4426 scope.go:122] "RemoveContainer" containerID="990d7b5af68d24b3bd3c7a4f29dc0182e33e6e9c8a6b8247d043d06ef976216c" Oct 31 04:42:05.202239 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:05.202204 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz"] Oct 31 04:42:05.206654 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:05.206615 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fp9bz"] Oct 31 04:42:05.309245 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:05.309211 4426 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k"] Oct 31 04:42:05.312995 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:05.312969 4426 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fxl7k"] Oct 31 04:42:07.128630 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:07.128598 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" path="/var/lib/kubelet/pods/4d600e53-d628-4f89-b6c6-186d3432e300/volumes" Oct 31 04:42:07.129069 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:07.129054 4426 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" path="/var/lib/kubelet/pods/76a0c679-9315-4024-a425-ae42d2716f0c/volumes" Oct 31 04:42:20.957235 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:20.957203 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-565fd9c896-fjbt8_47b46629-173e-4e33-b7a0-62236a5d7251/manager/0.log" Oct 31 04:42:21.464324 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:21.464287 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-565fd9c896-fjbt8_47b46629-173e-4e33-b7a0-62236a5d7251/manager/0.log" Oct 31 04:42:27.293314 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:27.293284 4426 scope.go:122] "RemoveContainer" containerID="2d8b20336cdc656a22e8bef23ab2dd84dac78833a876db4f3d5491a3550c020d" Oct 31 04:42:27.302069 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:27.302048 4426 scope.go:122] "RemoveContainer" containerID="31d773fd7c788324d72a8e052fc7f189b407bae9fa74250a4e877a9b67903234" Oct 31 04:42:27.310270 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:27.310241 4426 scope.go:122] "RemoveContainer" containerID="74f254d0d4c8bfa007b8bef4ea58a1526bb1ba3619725a89daf2208e96d70de0" Oct 31 04:42:32.210476 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.210448 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-97mff_e44d8d56-e743-4335-a287-2330fdeeb3a1/payload-processing/0.log" Oct 31 04:42:32.430879 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.430846 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-97mff_e44d8d56-e743-4335-a287-2330fdeeb3a1/payload-processing/0.log" Oct 31 04:42:32.790373 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.790339 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xddgj/must-gather-p5xp8"] Oct 31 04:42:32.790867 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.790847 4426 cpu_manager.go:414] "containerMap: removing container" podUID="d181a6d5-b87e-4f8c-8774-0ed5df832c04" containerName="revoke-keys" Oct 31 04:42:32.790962 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.790869 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d181a6d5-b87e-4f8c-8774-0ed5df832c04" containerName="revoke-keys" Oct 31 04:42:32.790962 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.790911 4426 cpu_manager.go:414] "containerMap: removing container" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" Oct 31 04:42:32.790962 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.790919 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" Oct 31 04:42:32.790962 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.790936 4426 cpu_manager.go:414] "containerMap: removing container" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" containerName="istio-proxy" Oct 31 04:42:32.790962 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.790944 4426 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" containerName="istio-proxy" Oct 31 04:42:32.791272 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.791048 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="d181a6d5-b87e-4f8c-8774-0ed5df832c04" containerName="revoke-keys" Oct 31 04:42:32.791272 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.791066 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a0c679-9315-4024-a425-ae42d2716f0c" containerName="istio-proxy" Oct 31 04:42:32.791272 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.791077 4426 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d600e53-d628-4f89-b6c6-186d3432e300" containerName="istio-proxy" Oct 31 04:42:32.796267 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.796245 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xddgj/must-gather-p5xp8" Oct 31 04:42:32.845873 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:42:32.845835 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:42:32.892269 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.892232 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xddgj\"/\"openshift-service-ca.crt\"" Oct 31 04:42:32.892456 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.892249 4426 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xddgj\"/\"kube-root-ca.crt\"" Oct 31 04:42:32.892456 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.892301 4426 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xddgj\"/\"default-dockercfg-x59w7\"" Oct 31 04:42:32.895020 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.895000 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xddgj/must-gather-p5xp8"] Oct 31 04:42:32.934942 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.934901 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whp7l\" (UniqueName: \"kubernetes.io/projected/3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a-kube-api-access-whp7l\") pod \"must-gather-p5xp8\" (UID: \"3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a\") " pod="openshift-must-gather-xddgj/must-gather-p5xp8" Oct 31 04:42:32.935158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:32.934996 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a-must-gather-output\") pod \"must-gather-p5xp8\" (UID: \"3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a\") " pod="openshift-must-gather-xddgj/must-gather-p5xp8" Oct 31 04:42:33.035996 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:33.035955 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a-must-gather-output\") pod \"must-gather-p5xp8\" (UID: \"3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a\") " pod="openshift-must-gather-xddgj/must-gather-p5xp8" Oct 31 04:42:33.036191 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:33.036050 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whp7l\" (UniqueName: \"kubernetes.io/projected/3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a-kube-api-access-whp7l\") pod \"must-gather-p5xp8\" (UID: \"3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a\") " pod="openshift-must-gather-xddgj/must-gather-p5xp8" Oct 31 04:42:33.036424 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:33.036402 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a-must-gather-output\") pod \"must-gather-p5xp8\" (UID: \"3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a\") " pod="openshift-must-gather-xddgj/must-gather-p5xp8" Oct 31 04:42:33.106433 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:33.106359 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whp7l\" (UniqueName: \"kubernetes.io/projected/3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a-kube-api-access-whp7l\") pod \"must-gather-p5xp8\" (UID: \"3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a\") " pod="openshift-must-gather-xddgj/must-gather-p5xp8" Oct 31 04:42:33.110274 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:33.110252 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xddgj/must-gather-p5xp8" Oct 31 04:42:33.483982 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:33.483946 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xddgj/must-gather-p5xp8"] Oct 31 04:42:33.490806 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:42:33.490780 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3878cbd5_4f6e_41a8_8e2c_4f9b2be6cd0a.slice/crio-10bf6f63b2bf9cfea770e59cd4e776e6525a54df86acce62ec74e6fc32e6755b WatchSource:0}: Error finding container 10bf6f63b2bf9cfea770e59cd4e776e6525a54df86acce62ec74e6fc32e6755b: Status 404 returned error can't find the container with id 10bf6f63b2bf9cfea770e59cd4e776e6525a54df86acce62ec74e6fc32e6755b Oct 31 04:42:33.492770 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:33.492751 4426 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:42:34.225737 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:34.225698 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xddgj/must-gather-p5xp8" event={"ID":"3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a","Type":"ContainerStarted","Data":"10bf6f63b2bf9cfea770e59cd4e776e6525a54df86acce62ec74e6fc32e6755b"} Oct 31 04:42:36.238294 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:36.238257 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xddgj/must-gather-p5xp8" event={"ID":"3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a","Type":"ContainerStarted","Data":"79a6b8bb7c44b045919588581a1775b2007f09ccd281348050b39a1906ee7690"} Oct 31 04:42:36.238687 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:36.238306 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xddgj/must-gather-p5xp8" event={"ID":"3878cbd5-4f6e-41a8-8e2c-4f9b2be6cd0a","Type":"ContainerStarted","Data":"d8ae7f834aba55f6b032977a046dfd5addae910229b215716252fe403f947ae2"} Oct 31 04:42:36.347641 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:36.347582 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xddgj/must-gather-p5xp8" podStartSLOduration=2.616012017 podStartE2EDuration="4.347566766s" totalImagesPullingTime="1.731554749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:42:36.346240398 +0000 UTC m=+4453.346865626" watchObservedRunningTime="2026-10-31 04:42:36.347566766 +0000 UTC m=+4453.348192017" Oct 31 04:42:44.337880 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:44.337831 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n9rx6_6e9ea6af-1755-4e15-861b-5740354972f2/global-pull-secret-syncer/0.log" Oct 31 04:42:48.513572 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:48.513534 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zmcjg_3dd3e64a-9a74-4542-b09a-7003e749b44f/konnectivity-agent/0.log" Oct 31 04:42:49.699338 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:49.699303 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-106.us-west-2.compute.internal_506bf97362686c30bca7e209bcab60a8/haproxy/0.log" Oct 31 04:42:53.305305 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:53.305261 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66fb56dd6d-wxqfb_2e9f99a4-7d1c-4783-9c5f-d565dd5f0931/authorino/0.log" Oct 31 04:42:53.618472 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:53.618376 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gc4kf_c79d294e-f5fc-4f02-bdda-5aa7885145bf/manager/0.log" Oct 31 04:42:54.012827 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.012799 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8gg9j_60f7506b-536a-4604-952d-d7447f9a3787/manager/0.log" Oct 31 04:42:54.658151 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.657733 4426 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp"] Oct 31 04:42:54.666382 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.666351 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.745163 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.745110 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-lib-modules\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.745434 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.745406 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgm7g\" (UniqueName: \"kubernetes.io/projected/e947f4b2-f26b-4e36-af6f-c0a25df9a876-kube-api-access-rgm7g\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.745559 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.745488 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-podres\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.745559 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.745535 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-sys\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.745647 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.745592 4426 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-proc\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.746592 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.746571 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp"] Oct 31 04:42:54.831021 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.830987 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7pknt_471df0d0-0980-416c-a410-dabeaeaa01ee/manager/0.log" Oct 31 04:42:54.846680 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.846640 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-proc\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.846869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.846802 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-proc\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.846869 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.846837 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-lib-modules\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.847001 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.846883 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgm7g\" (UniqueName: \"kubernetes.io/projected/e947f4b2-f26b-4e36-af6f-c0a25df9a876-kube-api-access-rgm7g\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.847001 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.846962 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-lib-modules\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.847001 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.846962 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-podres\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.847158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.847019 4426 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-sys\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.847158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.847029 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-podres\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.847158 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.847111 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e947f4b2-f26b-4e36-af6f-c0a25df9a876-sys\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.904889 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.904855 4426 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgm7g\" (UniqueName: \"kubernetes.io/projected/e947f4b2-f26b-4e36-af6f-c0a25df9a876-kube-api-access-rgm7g\") pod \"perf-node-gather-daemonset-dcrsp\" (UID: \"e947f4b2-f26b-4e36-af6f-c0a25df9a876\") " pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:54.986229 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:54.986106 4426 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:55.202692 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:55.202654 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-22j29_155f33ed-b4f9-4914-934d-82b04dcba3a1/limitador/0.log" Oct 31 04:42:55.365813 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:55.365726 4426 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp"] Oct 31 04:42:55.372876 ip-10-0-134-106 kubenswrapper[4426]: W1031 04:42:55.372844 4426 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode947f4b2_f26b_4e36_af6f_c0a25df9a876.slice/crio-220bfad0cc00d773e3f8fdd76e2a5349a86834f660b0912dd007c74668a10dbc WatchSource:0}: Error finding container 220bfad0cc00d773e3f8fdd76e2a5349a86834f660b0912dd007c74668a10dbc: Status 404 returned error can't find the container with id 220bfad0cc00d773e3f8fdd76e2a5349a86834f660b0912dd007c74668a10dbc Oct 31 04:42:55.427199 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:55.427164 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" event={"ID":"e947f4b2-f26b-4e36-af6f-c0a25df9a876","Type":"ContainerStarted","Data":"220bfad0cc00d773e3f8fdd76e2a5349a86834f660b0912dd007c74668a10dbc"} Oct 31 04:42:55.629528 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:55.629443 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2dljn_7a80e800-84ce-434c-98f6-f0e8823ed5d7/manager/0.log" Oct 31 04:42:56.434968 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:56.434922 4426 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" event={"ID":"e947f4b2-f26b-4e36-af6f-c0a25df9a876","Type":"ContainerStarted","Data":"8555d66c32f74308a6e466dda2f60f56fdfcc16956376725a710659f039673bf"} Oct 31 04:42:56.435633 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:56.435071 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:42:56.543943 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:42:56.543881 4426 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" podStartSLOduration=2.54386423 podStartE2EDuration="2.54386423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:42:56.543622334 +0000 UTC m=+4473.544247568" watchObservedRunningTime="2026-10-31 04:42:56.54386423 +0000 UTC m=+4473.544489458" Oct 31 04:43:02.451031 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:43:02.451001 4426 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xddgj/perf-node-gather-daemonset-dcrsp" Oct 31 04:43:02.824197 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:43:02.824078 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-q8wfl_a3beef8c-8001-47b8-a915-846eecb48f2d/monitoring-plugin/0.log" Oct 31 04:43:02.847532 ip-10-0-134-106 kubenswrapper[4426]: E1031 04:43:02.847471 4426 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lhsn4" containerName="extractor" Oct 31 04:43:05.575234 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:43:05.575200 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wbwhr_4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4/node-exporter/0.log" Oct 31 04:43:05.969758 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:43:05.969728 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wbwhr_4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4/kube-rbac-proxy/0.log" Oct 31 04:43:06.361829 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:43:06.361751 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wbwhr_4b381ebb-c6f2-44e2-8ee6-de22fe5d79c4/init-textfile/0.log" Oct 31 04:43:11.473925 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:43:11.473882 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-kmdh7_4d6bf979-c514-43c6-a329-98ae2336711d/prometheus-operator-admission-webhook/0.log" Oct 31 04:43:11.869730 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:43:11.869653 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bcf86fdd-bl8wn_2db970dc-067c-40d3-b333-031fd723eefe/telemeter-client/0.log" Oct 31 04:43:12.258980 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:43:12.258953 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bcf86fdd-bl8wn_2db970dc-067c-40d3-b333-031fd723eefe/reload/0.log" Oct 31 04:43:12.657508 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:43:12.657475 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bcf86fdd-bl8wn_2db970dc-067c-40d3-b333-031fd723eefe/kube-rbac-proxy/0.log" Oct 31 04:43:17.464516 ip-10-0-134-106 kubenswrapper[4426]: I1031 04:43:17.464488 4426 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d45d5c789lxlc9_16b2427e-bf1e-4892-ad55-9d5f2b93849b/istio-proxy/0.log"